Upgrade Weblate from git 2.18 version to latest docker

MAKE A BACKUP FIRST.
Ensure the website runs well before upgrading.

If encountering error ImproperlyConfigured: Requested setting DATA_DIR, but settings are not configured. You must either define the environment variable DJANGO_SETTINGS_MODULE or call settings.configure() before accessing settings. when starting server, run export DJANGO_SETTINGS_MODULE=weblate.settings.

Read Generic upgrade instructions from official weblate doc first.

Special instruction for my build: Custom changes to weblate (weblate/trans/formats.py)and translate-toolkit (translate/storage/properties.py)to provide a special “Minecraft Lang File” format. The HTTP server is Gunicorn with Nginx. The operating system is Ubuntu 16.04 and weblate runs on Python 2.7. The database is PostgreSQL 9.5. These softwares will be upgraded or replaced during progress.

Universal step:

Run git fetch --all before git operation to ensure latest code exists.

  1. Read Version specific instructions and follow it.
  2. Stop gunicorn/wsgi and celery worker.
  3. git stash (before 3.0.x)
  4. git reset --hard (before 3.0.x)
  5. git checkout weblate-x.x Usex.x.1 or the latest minor version if it exists for a major version.
  6. git stash pop (before 3.0.x)
  7. pip install -r requirements.txt
  8. find weblate -name '*.pyc' -delete to remove cached compiled bytecode.
  9. Update settings.py to accommodate to settings_example.py.
  10. python manage.py migrate --noinput
  11. follow other manage commands in generic upgrade instructions.
  12. Start the server and check if everything works well.

Version specified steps:

2.18 to 2.19
Nothing special

2.19 to 2.20
Update translate-toolkit to stable/2.3.x and apply the patch before installing weblate requirements.
Missing backports.csv, install it manually.

2.20 to 3.0
BACKUP BEFORE UPGRADING.
pg_dump weblate -U weblate -F t -f 2.20_py27_9.5.tar
Follow special steps in official docs.
From this version, changes to weblate and translate-toolkit is extracted to a single external file. Add the file to PYTHONPATH to let the weblate find it. To let weblate know the “Minecraft Lang File” format, add class path to WEBLATE_FORMATS in settings.py.
Change line 11 in file weblate/trans/migrations/0140_change_project.py from for change in Change.objects.all(): to for change in Change.objects.all().iterator():, or the migration progress will eat an enormous amount of memory, and won’t finish the job, really. Ref: https://www.azavea.com/blog/2017/02/23/strategies-reducing-memory-usage-django-migrations/

3.0 to 3.1
After installing requirements, jellyfish==0.8.1 will be installed, which is not compatible with python 2.7 because of using from itertools import zip_longest. To solve this, install jellyfish==0.6.1 mannually.
Change 'ENGINE': 'django.db.backends.postgresql_psycopg2' to 'ENGINE': 'django.db.backends.postgresql' in DATABASES in settings.py.

3.1 to 3.2
Weblate provided a new base class PropertiesBaseFormat for Java Properties file format. Change the “Minecraft Lang File” format class extending this class.
Start Celery workers manually by celery worker --app weblate --loglevel info --beat

3.3 to 3.4
Start Celery workers manually by ./examples/celery start.
If encountering error: OSError: [Errno 2] No such file or directory: '' when starting Celery worker, downgrade to celery==4.0 and kombu==4.0. Ref: https://github.com/WeblateOrg/weblate/issues/4029

3.6 to 3.7
Run apt install -y pkg-config libcairo2-dev python3-gi python3-gi-cairo gir1.2-gtk-3.0 libgirepository1.0-dev before installing requirements.

3.7 to 3.8
Upgrade six, pip, setuptools before installing requirements, or installing diff-match-patch==20181111 might lead to an error.

3.8 to 3.9
Default setting_example.py use Redis as cache and Celery broker. Start the Redis server before starting the server.

3.9 to 3.10
Celery workers eats memory. To reduce Celery memory usage, change --concurrency parameter to 1. E.g. edit ./weblate/examples/celery file.

3.11 to 4.0
BACKUP BEFORE UPGRADING.
4.0 needs Python 3.6 or above to run. Upgrade to Ubuntu 18.04 to get system-wide Python 3.6. apt update && apt upgrade , reboot, do-release-upgrade. Use pg_dropcluster and pg_upgradecluseter to migrate to PostgreSQL 10.
Weblate needs to be a superuser of PostgreSQL. To do this, su postgres and enter psql cli. Run ALTER USER weblate WITH SUPERUSER, and run \du to check if it works.
If encountering error: FileNotFoundError: [Errno 2] No such file or directory: ” when starting Celery worker again, instead of downgrade Celery, just add $PWD/ after file= in examples/celery. Ref: https://github.com/WeblateOrg/weblate/issues/4029
Remove old Python 2 packages like gunicorn to avoid running Python 2 script accidentally.
Clear Redis by run command redis-cli flushall or the new version of weblate might not understand. Clear data/avatar_cache if the avatar doesn’t show.

Dockerize Weblate

Follow official instruction: https://docs.weblate.org/en/weblate-4.1.1/admin/install/docker.html

  1. Start an empty weblate docker-compose instance and make sure it runs.
  2. Change docker-compose.override.yml to fit old settings.
  3. Upgrade PostgreSQL to the docker version.
  4. Replace weblate files and database to migrate.

Change docker-compose.override.yml

You may need to assign another admin email to avoid errors like CommandError: Multiple users matched given parameters!

Replace weblate files

Copy vcs directory is enough for migration. Ref: https://docs.weblate.org/en/weblate-4.1.1/admin/install.html#migrating-weblate-to-another-server
Write WEBLATE_FORMATS to /app/data/settings-override.py. Ref: https://docs.weblate.org/en/weblate-4.1.1/admin/install/docker.html#custom-configuration-files
Copy files to support “Minecraft Lang File” format to /app/data/python/ to let weblate find it. Ref: https://docs.weblate.org/en/weblate-4.1.1/admin/install/docker.html#adding-own-python-modules

Replace database

Use pg_dump to save old database, and use pg_restore to load.
Drop all tables in database before restore. Ref: https://stackoverflow.com/a/21247009

Version specified steps after dockerize

4.1 to 4.2
Update docker-compose repo. https://github.com/WeblateOrg/docker-compose
PostgreSQL has been upgraded to 12. To upgrade, using a repo suggested in the issue: https://github.com/docker-library/postgres/issues/37#issuecomment-221441743
Follow steps in readme https://github.com/tianon/docker-postgres-upgrade/blob/master/README.md
Prepare folders to save data. Use docker cp to copy all files from PostgreSQL volume to the old data directory, and create an empty directory for new data.
If anything went wrong, clear the new directory before the next try.
Use weblate username to upgrade by run docker run --rm -e PGUSER=weblate -e POSTGRES_INITDB_ARGS="-U weblate" -v DIR:/var/lib/postgresql tianon/postgres-upgrade:11-to-12 --link
Ref: https://github.com/tianon/docker-postgres-upgrade/issues/10#issuecomment-523020113
Note that DIR must be an absolute path.
After upgrading, modify pg_hba.conf or copy this file from the old data directory.
Ref: https://github.com/tianon/docker-postgres-upgrade/issues/16#issuecomment-570917033
Replace PostgreSQL volume content with upgraded data files in new data directory, and pull new version PostgreSQL docker image to finish.
Another issue is weblate is enforcing case insensitive uniqueness for username. Having two user accounts has the same username but case will lead to migration failure. An ideal solution is not possible. Just change one of the account’s username to another as a workaround.
Start PostgreSQL container only by run docker container start weblate-docker_cache_1. Run docker exec -it weblate-docker_database_1 psql -U weblate to log in PostgreSQL cli. To change the username of a certain user, run UPDATE weblate_auth_user set username = 'NewUsername' where username = 'OldUsername';.

Konsole on Windows (WSL)

Konsole的优点(vs XShell/Windows Terminal):

可以随意使用我习惯的 tmux 切 window/pane 快捷键: Shift+←→/Alt+↑↓←→ (XShell 中不可用)

开 tab 连 ssh 时, 能在 tab 上显示 ssh host, 便于区分

Tab title in Konsole
Tab title in Windows Terminal

没有以下奇怪问题

Windows Terminal 方块光标覆盖文字https://github.com/microsoft/terminal/issues/1203, 在 nvim 里仍然无效

复制完以后会取消选择 https://github.com/microsoft/terminal/issues/3884

python tqdm 中的进度条看起来正常(XShell 需要逐个将 属性-终端-将含糊的大小字符视为宽字符 的勾选去掉)

在 Windows 上安装使用 Konsole

基本思路是在 WSL(Windows Subsystem for Linux) 上跑 Konsole, 然后通过 X 转发, 在本地开一个 X Server 来显示 Konsole.

安装 WSL(Windows Subsystem for Linux)

我这里由于 Windows 10 没有 2004升级, 只能使用 WSL1, 直接在 Microsoft Store 里搜索发行版并安装即可. 由于我主要是用于 SSH, 基本与发行版无关, 就直接安装了 Ubuntu (会安装20.04 LTS).

安装配置一些常用软件, 比如 git zsh 等. oh-my-zsh 也可以直接安装上. 顺便安装一下 powerline 的字体 sudo apt install fonts-powerline

发现 htop 无法直接使用 https://github.com/microsoft/WSL/issues/25

安装 Konsole

直接用包管理器安装就好 sudo apt install konsole

配置 X 转发

我使用的 Windows X Server 是 vcxsrv. 安装完之后开始菜单里 XLaunch 一路默认启动即可.

执行 export DISPLAY=:0.0 即可, 添加到 .zshrc 方便使用.

ref: https://medium.com/@dhanar.santika/installing-wsl-with-gui-using-vcxsrv-6f307e96fac0

启动 Konsole

在 Ubuntu 里输入 konsole 并回车, 不出意外的话, Konsole 窗口就出现了.

图标丢失问题

此时应该会发现, 不少图标都丢失了. 比如 tab 的关闭按钮, 设置里左侧栏里的图标. 这应该没有安装提供这些图标的包导致的. 由于我还想用 Dolphin 等 KDE 软件, 我直接安装了 KDE 全家桶.

sudo apt install kde-full

安装完之后图标就正常了.

Dolphin 启动不起来

在 Ubuntu 里输入 dolphin 并回车, 提示

dolphin: error while loading shared libraries: libQt5Core.so.5: cannot open shared object file: No such file or directory

解决方法: 执行 sudo strip --remove-section=.note.ABI-tag /usr/lib/x86_64-linux-gnu/libQt5Core.so.5 即可

ref: https://superuser.com/questions/1347723/arch-on-wsl-libqt5core-so-5-not-found-despite-being-installed#comment2229452_1348051

鼠标指针被裁剪问题

此时会发现, 鼠标指针都被裁剪掉了一部分

cropped pointer cursor
cropped text cursor

似乎是老问题. 暂时没有解决, 通过缩小鼠标指针来提高体验.

执行 export XCURSOR_SIZE=32 即可, 也放进 .zshrc 方便使用. 修改为32后, 仍然有部分被裁剪掉了, 不过现在被裁剪的部分较少, 而且尺寸与我使用的 Windows 的鼠标指针大小基本相同, 姑且可以接受.

small pointer cursor
small text cursor

让 Konsole 更 Windows

此时打开 Konsole 的流程需要从开始菜单打开 Ubuntu, 输入 konsole 回车. 而 Ubuntu 的窗口还会开着. 这很不 Windows.

使用以下 vbs 脚本来启动 Konsole

Set oShell = CreateObject ("Wscript.Shell")
Dim strArgs
strArgs = "wsl zsh -c 'DISPLAY=localhost:0.0 XCURSOR_SIZE=32 konsole --workdir /home/epix'"
oShell.Run strArgs, 0, false

然后在 %AppData%\Microsoft\Windows\Start Menu 创建个快捷方式, 图标改成 Konsole 的图标, 就可以直接从开始菜单打开 Konsole 了.

ref: https://github.com/microsoft/WSL/issues/3404#issuecomment-408094942

仍然存在的问题

开启速度不如原生, 连接 ssh 时花费时间更多.

一些可能能够提高 SSH 体验的方法

将常用的 ssh 配置写入 .ssh/config 中, 用到时只要 ssh <Host字段> 即可.

使用 ssh-agent 来缓存解密的 ssh private key, 省去每次输入的麻烦, 方法

  1. 可以直接运行 eval `ssh-agent` 来启动 ssh-agent. 不能只运行 ssh-agent 的原因: ssh-agent 会把启动后需要执行的设置环境变量等信息输出, 需要 eval 后生效. ref: https://stackoverflow.com/questions/17846529/could-not-open-a-connection-to-your-authentication-agent/4086756#4086756
  2. 可以手动指定 ssh-agent 监听地址, 然后设置环境变量. 比如
    ssh-agent -a /run/shm/ssh-agent.sock export SSH_AUTH_SOCK=/run/shm/ssh-agent.sock

我使用的是方法2, 并将此内容写入 .zshrc, 每次进 zsh 都会尝试开 ssh-agent (已启动则会启动失败), 并设置环境变量.

ssh-agent 启动并设置好环境变量之后, 可以通过以下方法缓存

  1. 手动执行 ssh-add <path to private key> , 输入密码即可
  2. .ssh/config 中设置 AddKeysToAgent yes, 解密后加入 ssh-agent. 可以放到 Host * 下(对所有 Host 生效) 也可以放在某个 Host 下(只对此 Host 生效) ref:

我使用的是方法2, 每次开机后第一次连接需要输入密码, 之后只要 ssh-agent 存活就不需要再输入. (Konsole 窗口关闭后不会关闭 ssh-agent)

双重 X 转发

此时 Konsole 已经是通过 X 转发显示的了. 如果想要再将这里面 ssh 到的远程机器的 GUI 通过 X 转发回本地, 还需要一些额外设置.

  1. WSL 中安装 xauth
  2. .ssh/config中需要 X 转发的 Host 下添加
ForwardAgent yes
ForwardX11 yes
ForwardX11Trusted yes
XauthLocation /usr/bin/xauth

ref: https://unix.stackexchange.com/a/393766

EhViewer MOD

我当前主力的看漫画 Android 平板是联想 MIIX 520 上的模拟器. 与正常平板有所不同, 比如长按被 Windows 劫持了.
为了提高 EhViewer 体验, 以及适配我的 Android 设备和我平时的阅读习惯(比如先下载再看, 倒序阅读), 对 EhViewer 进行了部分修改.
由于修改完全依照个人喜好, 暂时无意发起 Pull Request.

修改内容:

  1. 在下载界面内, 右上角菜单增加若干按钮, 分别是: 全选, 进入选择模式, 选择已选择条目及以下所有条目, 选择已读条目(阅读进度至少2页), 倒序开始下载全部, 选择已下载条目.
  2. 在画廊列表界面内, 每个条目上增加一个按钮, 点击可以直接添加到下载, 而无需打开详情再下载. 如果条目已经在下载列表内, 或者已经阅读过, 则按钮隐藏.
  3. 在下载/画廊列表界面, 显示 阅读进度/总页数, 并且使用渐变颜色指示进度, 完全没读过为红色, 读完为绿色.
  4. 阅读时, 滑动阅读进度条, 不需要松手就直接显示进度条指示的页面.
  5. 如果出现509或者IP ban, 停止所有下载.
  6. 添加新的三方tag翻译源.
  7. 允许在每张图片下载直接延迟一会儿, 以防止网速过快导致被 IP ban.

源码:
https://github.com/exzhawk/EhViewer
下载:
https://github.com/exzhawk/EhViewer/releases

注意, 下载的 APK 签名与原版不同, 可能需要先卸载官方版. 可以通过 app 内置的导出功能: 官方版内导出数据, 卸载官方版, 安装 mod 版, mod 版内导入数据. 也可以直接使用钛备份等一般方法保留数据.

截图

How many trailing zeros does a factorial have

Simple question: How many trailing zero does 10000! have?

Simple to solve it using python

import math

factorial_answer = str(math.factorial(10000))
print(len(factorial_answer) - len(factorial_answer.rstrip('0')))

Python think about 100ms and tell me the answer is 2499.

This question is name as “Trailing zeros”
https://en.wikipedia.org/wiki/Trailing_zeros

The question become:

So we can just integer divide 10000 by 5 and get 2000, integer divide 2000 by 5 and get 400…. and sum all answers (2000, 400, …) until get 0.

let’s using recursion

def f(r, c=0):
    if r == 0:
        return c
    else:
        return f(r // 5, c + r // 5)


print(f(10000))

The function take two arguments, one for current answer, another for count result.

and make it into one line

f = lambda r, c=0: c if r == 0 else f(r // 5, c + r // 5)
print(f(10000))

Here we assign lambda expression to f and invoke it recursively. However, assign lambda expression is not recommended in PEP8. 不, 这不PEP8. And function f is exposed in namespace.  There are ways to make recursive into anonymous lambda function that won’t pollute namespace, as well as make it into really one line.

print((lambda a: lambda r, c = 0: a(a, r, c))(lambda s, r, c: c if r == 0 else s(s, r // 5, c + r // 5))(10000))

Another flaw is, we use same division twice, more precisely, “r // 5” twice.

To eliminate duplicate, closure may help.

def f(r):
    def _f(r):
        while r != 0:
            r //= 5
            yield r

    return sum(_f(r))


print(f(10000))

The process become, constantly dividing number by 5 and yield it to sum, until zero.

Make a function for that “continuously do something and yield it, until some condition” operation

def repeat_w_func(initial_input, repeat_func=None, yield_input=False, end_func=None):
    """
    constantly do calculate function and take output as next time input
    :param initial_input: the initial input
    :param repeat_func: the function that will be evaluate repeatedly
    if not set, the function behavior like itertools.repeat(initial_input)
    :param yield_input: if set to True, initial_input will be yield once before yielding repeat_func output
    :param end_func: the function that will end the loop and stop yielding. if not set, yielding will be endless
    :return: a generator
    """
    result = initial_input
    if yield_input:
        yield result
    if repeat_func:
        while True:
            if end_func:
                if end_func(result):
                    break
            result = repeat_func(result)
            yield result

    else:
        while True:
            yield initial_input

With brand new repeat with function function, we can solve the question elegantly, with help of itertools.takewhile function.

from itertools import takewhile
print(sum(takewhile(lambda x: x != 0, repeat_w_func(10000, lambda x: x // 5))))

Future research

Solve in function programming language like Haskell.

Is there a written functional just work like the repeat_w_func function in python or package?

Install Keras with CUDA on Windows 10 PC

Objective

The objective of this post is guide you use Keras with CUDA on your Windows 10 PC.

Install Dependencies

Hardware: A graphic card from NVIDIA that support CUDA, of course.

Driver: Download and install the latest driver from NVIDIA or your OEM website

CUDA: Download and install version 7.5 from  https://developer.nvidia.com/cuda-75-downloads-archive DO NOT install outdated driver intergrated in installer. Visual Studio integration and GPU Deployment Kit is not needed.

cuDNN: Download and copy all folder (there should be three: “bin”, “include”, “lib”) in zipped cuda folder to your CUDA installation folder (there should be “bin”, “include”, “lib”) https://developer.nvidia.com/cudnn You may need to register as an NVIDIA Accelerated Computing Developer Program member to process to download.

Python: Download and install Anaconda 2 from https://www.continuum.io/downloads Run conda install mingw libpython after installation.

Visual Studio: Install version 2010, 2012, or 2013. Newer version is not supported. The free community version is fine. We just need the compiler.

Microsoft Visual C++ Compiler for Python 2.7: Download and install from http://www.microsoft.com/en-us/download/details.aspx?id=44266

GCC: Download and install TDM-GCC from http://tdm-gcc.tdragon.net/

Install Theano and Keras

Keras support Theano or Tensor Flow as backend. However, Tensor Flow with GPU is not support in Windows. So just use Theano as backend.

Just install as a common package of python
pip install theano keras

Configuring Theano

Write a plain text file named .theanorc (or .theanorc.txt if previous one is hard to create)in your user folder(C:\Users\<Your username here>\)

Contents is as following

[global]
floatX = float32
device = gpu

[nvcc]
flags=-LC:\Apps\Anaconda2
compiler_bindir=C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin

[dnn]
enabled = True

[lib]
cnmem=0.75

“device = gpu” means use gpu resource
Replace “C:\Apps\Anaconda2” to your installation path of your Anaconda 2. There is no space between “flags=-L” and the path.

Replace “C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin” to your installation path of Visual Studio. There should be a “cl.exe” at that path.

Replace “cnmem=0.75” to your free graphic card memory ratio. You may use GPU-Z from https://www.techpowerup.com/gpuz/ to determine how much memory can be allocated to Theano besides your normal usage (for screen display, etc.). Theano will eat that much memory (75% of memory in this configuration) when you initialize Theano every time.

Test Theano

Open python console and run import theano

It may be a little slow, but eventually it will print something like

Using gpu device 0: <Your GPU> (CNMeM is enabled with initial size: <Your cnmem ratio> of memory, cuDNN <numbers, whatever>)

Stop python console if it looks fine.

Configuring Keras

To manually assign Theano backend, change following lines in C:\Users\<Your username here>\.keras\keras.json

{
    "image_dim_ordering": "tf", 
    "epsilon": 1e-07, 
    "floatx": "float32", 
    "backend": "theano"
}

Replace "image_dim_ordering": "tf", to "image_dim_ordering": "th", to use Theano’s image channel order(BGR).

Replace "backend": "tensorflow" to "backend": "theano" to assign Theano as backend.

Test Keras

Just run the example from official repositry https://github.com/fchollet/keras/blob/master/examples/imdb_cnn.py

Wait and you will know if it works well. You may also watch GPU-Z to know how many GPU resource is been used.

Conclusion

Now you get a fully workable Keras instance with CUDA acceleration.

PotPlayer Mini Skin Fixed for version 1.6.59347

After an update of PotPlayer, the older mini skin seems stopped working. Only a windows style border w/ title bar are shown with no control components nor progress bar. But modified mini skin on DA still working. So I wonder what’s the difference.

Analysis: PotPlayer skin file has ‘.dsf’ extension. Opening in hex editor, it has file head of “PK”. Obviously it’s ZIP. After extracting, it’s “VideoSkin.xml” that define the skin appearance. It seems that “VideoSkin.xml” of the original mini skin is some kind of broken or in wrong format which new version of PotPlayer can’t parse.

Fix: Correct hard-coded string in “VideoSkin.xml” and save as UTF-8(w/o BOM) and re-package into ZIP file.

Download: https://github.com/exzhawk/Mini_UTF8.dsf/releases