Tag: python

Posted on

pycryptco in python3

最近需要做 public/private key 的加解密，所以找到 pycryptco 。

使用的例子可以參考 Python and cryptography with pycrypto

但使用時，卻遇到 unsupported operand type(s) for pow() 的問題，經過一番查找，才找到問題所在，詳情可以看這篇：Error with encrypt message with RSA python – Stack Overflow

主要是因為 Python 3 的字串預設是 unicode 字串，而 encrypt 沒辦法處理，所以得先 encode 為 utf-8 字串才行：

# -*- coding: utf-8 -*-
import os
import base64
from Crypto.PublicKey import RSA
from Crypto import Random
from Crypto.Cipher import PKCS1_OAEP, AES
from pprint import pprint


# Generate key
if os.path.exists('mykey.pem'):
    with open('mykey.pem', 'rb') as fin:
        key = RSA.importKey(fin.read())
else:
    random_generator = Random.new().read
    key = RSA.generate(1024, random_generator)
    with open('mykey.pem', 'wb') as fout:
        fout.write(key.exportKey('PEM'))

# Display key information
print("key = {}".format(key))
print("can_encrypt = {}".format(key.can_encrypt()))
print("can_sign = {}".format(key.can_sign()))
print("can_private = {}".format(key.has_private()))

# show public key
pprint(dir(key.publickey()))

# Encrypt
enc_data = key.publickey().encrypt("abcdef".encode('utf-8'), 32)
pprint(enc_data)

# Decrypt
print(key.decrypt(enc_data))
Posted on

Ubuntu trusty python2 與 python3 的 PYTHONPATH

python2 的 sys.path 是

['',
 '/usr/local/lib/python2.7/dist-packages/python_nghttp2-1.3.5.dev0-py2.7-linux-x86_64.egg',
 '/usr/lib/python2.7',
 '/usr/lib/python2.7/plat-x86_64-linux-gnu',
 '/usr/lib/python2.7/lib-tk',
 '/usr/lib/python2.7/lib-old',
 '/usr/lib/python2.7/lib-dynload',
 '/home/ellery/.local/lib/python2.7/site-packages',
 '/usr/local/lib/python2.7/dist-packages',
 '/usr/lib/python2.7/dist-packages',
 '/usr/lib/python2.7/dist-packages/PILcompat',
 '/usr/lib/python2.7/dist-packages/gtk-2.0',
 '/usr/lib/pymodules/python2.7',
 '/usr/lib/python2.7/dist-packages/ubuntu-sso-client',
 '/usr/lib/python2.7/dist-packages/wx-2.8-gtk2-unicode']

 

python3 的 sys.path 是

['',
 '/usr/local/lib/python3.4/dist-packages/onedrive_d-1.1.0dev-py3.4.egg',
 '/usr/local/lib/python3.4/dist-packages/daemonocle-0.8-py3.4.egg',
 '/usr/local/lib/python3.4/dist-packages/Send2Trash-1.3.0-py3.4.egg',
 '/usr/local/lib/python3.4/dist-packages/certifi-2015.11.20.1-py3.4.egg',
 '/usr/local/lib/python3.4/dist-packages/psutil-2.1.1-py3.4-linux-x86_64.egg',
 '/usr/lib/python3/dist-packages',
 '/usr/lib/python3.4',
 '/usr/lib/python3.4/plat-x86_64-linux-gnu',
 '/usr/lib/python3.4/lib-dynload',
 '/home/ellery/.local/lib/python3.4/site-packages',
 '/usr/local/lib/python3.4/dist-packages']

 

會發現兩邊的 $HOME/.local/lib/pythonX.Y/site-packages 順序是不一樣的，照理來說，應該是要以 $HOME/.local/lib/pythonX.Y/site-packages 為優先，這樣才有機會安裝最新版本的 package 而無需更動到系統的 package。

那該怎麼辦呢？這可以修改 /etc/python3.4/sitecustomize.py 來解決這問題。

在 /etc/python3.4/sitecustomize.py 的最後加入

import os
import sys

sys.path.insert(0,
        os.path.join(
            os.path.expanduser("~"),
            ".local",
            "lib",
            "python3.4",
            "site-packages"))

Python3 在執行前，會先執行這個檔案，這裡在 sys.path 串列最前面插入 $HOME/.local/lib/pythonX.Y/site-packages ，就能解決這問題了。

參考資料：

Posted on

Python on Visual Studio Code 初步設定

Visual Studio Code 有獨立一個頁面介紹 Python 的部份：Python with Visual Studio Code

基本有4個 extension 要裝：

  • Python
  • Code Runner
  • MagicPython
  • Python for VSCode

Python 這個 extension 可以設定的東西很多，設定可以參考它的 Wiki ，可是知道有這些設定以後，該怎麼設定呢？你得先在專案資料夾下建立一個 .vscode 的資料夾，然後在裏面新增一個 settings.json ，這邊再參考 Wiki 去填入設定就可以了。

目前我的 Django 專案用的設定是這樣：

{
    "python.pythonPath": "/home/user/.virtualenvs/myproject/bin/python",
    "python.linting.flake8Enabled": true,
    "python.formatting.provider": "autopep8",
    "name": "Django",
    "type": "python",
    "request": "launch",
    "stopOnEntry": false,
    "program": "${workspaceRoot}/manage.py",
    "args": [
        "runserver",
        "--noreload"
    ],
    "debugOptions": [
        "WaitOnAbnormalExit",
        "WaitOnNormalExit",
        "RedirectOutput",
        "DjangoDebugging"
    ]
}

 

因為我用的是 virtualenv ，所以 python 執行檔跟相關的 library 都是在 $HOME/.virtualenv 這裡，把 python.pythonPath 設到這裡以後， intellisense/autocomplete 就可以正常運作。此外我也加上了 linting 與執行、除錯的相關設定。

其他還有 Jupyter 等等的，還沒去深入了解，再找時間來看。

Posted on

升級 daphne 到 1.2.0 後導致連線卡住

使用 django channels 得使用 daphne 來處理 asgi (websocket/long poll http) 請求，我遇到的情況是從 1.1.0 升級到 1.2.0 以後，整個 django 應用就無法接受服務。

查了很久，看程式也覺得應該沒問題，該處理的 request ，daphne 都有轉送給 channels 去處理，而 channels 也都有處理。心裡想，這應該是個大問題，早該有人回報吧。但用 google 查詢都沒查到，今天心血來潮查了 Daphne 在 github 上的 closed issues，總算是找到了：daphne==1.2.0 hangs forever · Issue #105 · django/daphne

解法也很簡單，升級 asgi_redis/asgi_ipc 到新版就可以了。下次升級 daphne 時，得注意。

Posted on

pipdeptree

前幾天上傳程式到 openshift 時，發現新版的 six 安裝不上去，原因是 OpenShift 所帶的 six 是 0.3 版，但是無法透過 sudo 等之類的方式來安裝新版，所以想查到底是哪個模組會需要用到新版的 six。上網找了一下，發現 pipdeptree 可以滿足我的需求。

用 pip install pipdeptree 就可以安裝，使用上也很簡單，輸入 pipdeptree 就會列出各個模組的相依性與其所需最低版本。

結果大致是這樣：

Warning!!! Possibly conflicting dependencies found:
* HaodooScraper-Flask==1.0
 - python-dateutil [required: ==2.3, installed: 2.6.0]
 - six [required: ==1.3.0, installed: 1.10.0]
------------------------------------------------------------------------
alembic==0.6.7
  - Mako [required: Any, installed: 1.0.0]
    - MarkupSafe [required: >=0.9.2, installed: 0.23]
  - SQLAlchemy [required: >=0.7.3, installed: 0.9.8]
cssselect==0.9.1
Flask-APIBlueprint==1.0.0
  - flask [required: >=0.11.1,<1.0, installed: 0.12] - click [required: >=2.0, installed: 6.7]
    - itsdangerous [required: >=0.21, installed: 0.24]
    - Jinja2 [required: >=2.4, installed: 2.9.5]
      - MarkupSafe [required: >=0.23, installed: 0.23]
    - Werkzeug [required: >=0.7, installed: 0.12.1]
  - six [required: <2.0,>=1.10.0, installed: 1.10.0]
HaodooScraper-Flask==1.0
  - Flask [required: Any, installed: 0.12]
    - click [required: >=2.0, installed: 6.7]
    - itsdangerous [required: >=0.21, installed: 0.24]
    - Jinja2 [required: >=2.4, installed: 2.9.5]
      - MarkupSafe [required: >=0.23, installed: 0.23]
    - Werkzeug [required: >=0.7, installed: 0.12.1]
  - Flask-Bootstrap [required: Any, installed: 3.3.7.1]
    - dominate [required: Any, installed: 2.3.1]
    - Flask [required: >=0.8, installed: 0.12]
      - click [required: >=2.0, installed: 6.7]
      - itsdangerous [required: >=0.21, installed: 0.24]
      - Jinja2 [required: >=2.4, installed: 2.9.5]
        - MarkupSafe [required: >=0.23, installed: 0.23]
      - Werkzeug [required: >=0.7, installed: 0.12.1]
    - visitor [required: Any, installed: 0.1.3]
  - Flask-JsonTools [required: Any, installed: 0.1.1.post0]
    - flask [required: >=0.10.1, installed: 0.12]
      - click [required: >=2.0, installed: 6.7]
      - itsdangerous [required: >=0.21, installed: 0.24]
      - Jinja2 [required: >=2.4, installed: 2.9.5]
        - MarkupSafe [required: >=0.23, installed: 0.23]
      - Werkzeug [required: >=0.7, installed: 0.12.1]
  - flask-restplus [required: Any, installed: 0.10.1]
    - aniso8601 [required: >=0.82, installed: 1.2.0]
      - python-dateutil [required: Any, installed: 2.6.0]
        - six [required: >=1.5, installed: 1.10.0]
    - Flask [required: >=0.8, installed: 0.12]
      - click [required: >=2.0, installed: 6.7]
      - itsdangerous [required: >=0.21, installed: 0.24]
      - Jinja2 [required: >=2.4, installed: 2.9.5]
        - MarkupSafe [required: >=0.23, installed: 0.23]
      - Werkzeug [required: >=0.7, installed: 0.12.1]
    - jsonschema [required: Any, installed: 2.6.0]
    - pytz [required: Any, installed: 2017.2]
    - six [required: >=1.3.0, installed: 1.10.0]
  - Flask-WTF [required: Any, installed: 0.14.2]
    - Flask [required: Any, installed: 0.12]
      - click [required: >=2.0, installed: 6.7]
      - itsdangerous [required: >=0.21, installed: 0.24]
      - Jinja2 [required: >=2.4, installed: 2.9.5]
        - MarkupSafe [required: >=0.23, installed: 0.23]
      - Werkzeug [required: >=0.7, installed: 0.12.1]
    - WTForms [required: Any, installed: 2.1]
  - lxml [required: Any, installed: 3.4.0]
  - python-dateutil [required: ==2.3, installed: 2.6.0]
    - six [required: >=1.5, installed: 1.10.0]
  - requests [required: Any, installed: 2.13.0]
  - six [required: ==1.3.0, installed: 1.10.0]
  - SQLAlchemy [required: Any, installed: 0.9.8]
nose==1.2.1
pipdeptree==0.10.1
  - pip [required: >=6.0.0, installed: 9.0.1]
pkg-resources==0.0.0
PyMySQL==0.6.2
pyOpenSSL==0.14
  - cryptography [required: >=0.2.1, installed: 1.8.1]
    - asn1crypto [required: >=0.21.0, installed: 0.22.0]
    - cffi [required: >=1.4.1, installed: 1.10.0]
      - pycparser [required: Any, installed: 2.17]
    - idna [required: >=2.1, installed: 2.5]
    - packaging [required: Any, installed: 16.8]
      - pyparsing [required: Any, installed: 2.2.0]
      - six [required: Any, installed: 1.10.0]
    - setuptools [required: >=11.3, installed: 34.3.3]
      - appdirs [required: >=1.4.0, installed: 1.4.3]
      - packaging [required: >=16.8, installed: 16.8]
        - pyparsing [required: Any, installed: 2.2.0]
        - six [required: Any, installed: 1.10.0]
      - six [required: >=1.6.0, installed: 1.10.0]
    - six [required: >=1.4.1, installed: 1.10.0]
  - six [required: >=1.5.2, installed: 1.10.0]
wheel==0.30.0a0
Posted on

Django queryset 對日期時間欄位的額外設定

Django queryset 對日期時間的處理已經很完備了，可以透過使用 __year 或 __month 等方式來找到是某年或某月的紀錄。

這兩天碰到的狀況是，資料是 MySQL 時，日期時間的比對 (__year / __month) 失效了。仔細看過文件以後，才發現 MySQL 需要事先設定，使用 mysql_tzinfo_to_sql 載入時區表格才行。

This function performs time zone conversions directly in the database. As a consequence, your database must be able to interpret the value of tzinfo.tzname(None). This translates into the following requirements:

在終端機 (shell) 裡，輸入下列指令：

mysql_tzinfo_to_sql /usr/share/zoneinfo | mysql -u root mysql

接著重新啟動 MySQL 伺服器即可。

mysql_tzinfo_to_sql 的用法不只一種，我選擇的是最簡單的用法。

Posted on

mypy

演講：https://youtu.be/ZP_QV4ccFHQ

mypy 是靜態型別檢查工具，遵循 PEP484 與 PEP526 這兩個標準

而 PEP484 與 PEP526 這兩個標準最主要的用途是為 Python 加上型別標示，這樣可以更容易找到潛在的錯誤。

安裝

pip3 install mypy-lang

程式的部份 (type annotation)

Python 3

def hello(name: str) -> str:
    """foo."""
    return "Hello, {}".format(name)


def main() -> None:
    """Main entry."""
    hello("John")

Python 2

def hello(name):
    # type: (str) -> str
    """foo."""
    return "Hello, {}".format(name)


def main():
    # type: () -> None
    """Main entry."""
    hello("John")

如何使用 mypy 檢查

如果程式沒加 type annotation，那麼執行 mypy 不會有任何問題。 如果有一部份使用了 type annotation，那麼執行 mypy 就會告知有問題。 要強制檢查，可以使用 –disallow-untyped-defs

mypy --disallow-untyped-defs your_program.py

要產生 HTML 報告

mkdir html
mypy --html-report ./html your_program.py
xdg-open ./html/index.html

產生 JUnit XML

mypy --junit-xml JUNIT_XML your_program.py

案例

Dropbox 內部已經在使用。

Posted on

django-channels websocket 與 nginx

django-channels 的佈署指南，裏面的幾個重點：

  1. 先安裝 redis，再用 pip 安裝 asgi_redis ，然後將 settings 裡 CHANNEL_LAYERS 的 “BACKEND” 改為 asgi_redis.RedisChannelLayer ，這樣效能會比較好些。
  2. 執行 worker ： 
    python manage.py runworker
  3. 在 wsgi.py 的同個資料夾，新增 asgi.py ，裏面放 
    import os
from channels.asgi import get_channel_layer

os.environ.setdefault("DJANGO_SETTINGS_MODULE", "my_project.settings")

channel_layer = get_channel_layer()
  4. 執行 ASGI server ： 
    daphne your_project.asgi:channel_layer

    ，daphne 是在安裝 channels 時，會跟著裝上的。使用了 daphne 以後，就可以不需要 gunicorn 了，至於 uwsgi ，因為我沒在用，所以沒有深入研究。

runworker 跟 daphne 的部份，可以使用 supervisor 或者是寫 upstart script ，讓他們自動啟動。除了這些重點以外，nginx 的設定也需要調整，設定不多，只有幾行，這可以參考這兩篇文章：

大致的設定如下：

map $http_upgrade $connection_upgrade {
    default upgrade;
    '' close;
}

upstream app_server {
    server 127.0.0.1:8000 fail_timeout=0;
}

server {
    include mime.types;
    default_type application/octet-stream;
    access_log /var/log/nginx/access.log combined;
    sendfile on;

    listen 80;
    client_max_body_size 4G;

    # set the correct host(s) for your site
    server_name example.com;

    keepalive_timeout 5;

    # path for static files
    location /static/ {
        alias /srv/app/site/static/;
    }

    location /media/ {
        alias /var/app/media/;
    }

    location / {
        # checks for static file, if not found proxy to app
        try_files $uri @proxy_to_app;
    }

    location @proxy_to_app {
        proxy_buffering off;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $http_host;
        proxy_redirect off;

        proxy_pass   http://app_server;

        # For websocket
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
    }

    error_page 500 502 503 504 /500.html;
    location = /500.html {
        root /srv/app/site/static;
    }
}

我在套用這些設定以後，websocket 仍然無法順利連上，找了好半天，才找到這篇：Websockets fail to work after upgrading to 1.0.0 · Issue #466 · django/channels  ，看完才知道，channels/daphne 在升級到 1.0 以後，程式裡的 websocket consumer 必須要送出 

{"accept": True}

才行。這部份可以參考 1.0 的 release note ，也可以參考新的 Getting Start。不過我在參考 Getting start 時，居然沒注意到這行，也是我太大意了。