Python 3 Support

[lxp]

Python 2 will be end-of-life in 2020 (https://www.python.org/dev/peps/pep-0373/#update), so we have to add support for Python 3.

[stevenxxiu]

Is Python 2 still needed? If not I can try port it.

[lxp]

I would be happy to have your help for porting to Python 3. The main goal is to have it running on Python 3. As a bonus it would be great, if it still runs on Python 2.7 with compatibility layers like python-future [1].

I already gave it a try last year using futurize [2] and manual fixes. I have now pushed my work in the python3 branch. You can use this as a starting point, if you want. However, the test cases do not pass yet.

To run the unit and integration tests, you can use the following commands:

~/cfv/test$ python3 test.py --unit --exit-early
*** testing all_unittests_suite
test_lchoplen_compose (test_strutil.chopTestCase) ... ok
test_lchoplen_simple (test_strutil.chopTestCase) ... ok
test_lchoplen_wide (test_strutil.chopTestCase) ... ok
test_rchoplen_compose (test_strutil.chopTestCase) ... ok
test_rchoplen_simple (test_strutil.chopTestCase) ... ok
test_rchoplen_wide (test_strutil.chopTestCase) ... ok
test_compose_noprecombined (test_strutil.uwidthTestCase) ... ok
test_halfwidth (test_strutil.uwidthTestCase) ... ok
test_nonspacing (test_strutil.uwidthTestCase) ... ok
test_simple (test_strutil.uwidthTestCase) ... ok
test_wide (test_strutil.uwidthTestCase) ... ok
test_get_path_key (test_caching.AbsPathKeyTest) ... ok
test_rename (test_caching.AbsPathKeyTest) ... ok
test_nocase_findfile (test_caching.RelPathKeyTest) ... ok
test_nocase_findfile_parent (test_caching.RelPathKeyTest) ... ok
path_split (cfv.osutil)
Doctest: cfv.osutil.path_split ... ok
strippath (cfv.osutil)
Doctest: cfv.osutil.strippath ... ok
lchoplen (cfv.strutil)
Doctest: cfv.strutil.lchoplen ... ok
rchoplen (cfv.strutil)
Doctest: cfv.strutil.rchoplen ... ok
human_int (benchmark)
Doctest: benchmark.human_int ... ok

----------------------------------------------------------------------
Ran 20 tests in 0.072s

OK
.OK (False)

~/cfv/test$ python3 test.py --exit-early
>>> testing...
...................
>>> failed test: ~/cfv/test/cfv -ZNVRMUI --unquote=no --fixpaths="" --strippaths=0 --showpaths=auto-relative --progress=no --announceurl=url --noprivate_torrent -r -p /tmp/tmpgudzgbaz --renameformat="%(name)s-%(count)i%(ext)s" -C -t sha512 
/tmp/tmpgudzgbaz/test.ext.end : undecodable filename
/tmp/tmpgudzgbaz/test2.foo : undecodable filename
/tmp/tmpgudzgbaz/test3 : undecodable filename
/tmp/tmpgudzgbaz/d2/test4.foo : undecodable filename
4 files, 0 OK, 4 file errors.  0.002 seconds, 0.0K/s
FAILED (16)
  File "test.py", line 1890, in <module>
    failed += all_tests()

  File "test.py", line 1636, in all_tests
    ren_test(fmt)

  File "test.py", line 914, in ren_test
    test_generic('%s -C -t %s' % (cmd, f), cfv_test)

  File "test.py", line 287, in test_generic
    test_log_results(cfvtest.cfvenv + cfvtest.cfvfn + ' ' + cmd, s, o, r, kw)

  File "test.py", line 272, in test_log_results
    test_log_finish(cmd, s, r, o, kw)

  File "test.py", line 255, in test_log_finish
    traceback_str = '\n'.join(traceback.format_stack())

[1] https://python-future.org/
[2] https://python-future.org/futurize.html

[stevenxxiu]

It's been quite tedious so far, and I reckon having it working with Python 3 is more than enough. Got most of the tests working now, just 32 left!

[stevenxxiu]

I'm not sure if I'll continue porting this however, the code doesn't look all that well written from reading it. Maybe I'm better off using other utilities.

But would be happy if you can continue where I left off.

[lxp]

Thanks for your effort! I am sad to hear that you stopped your porting work for now. However, I think you already got a lot further with your port, than my last try. I know the code base is old and not easy to port. Did you find any major show stopper that requires a lot of re-work?

I do not know about any similar tool with that many features, that's why I am trying to keep cfv alive. If you find some other tool with a similar feature-set, I am happy to hear about it myself.

I will give your branch a try, once time allows it.
If you decide to continue your work, you are welcome of course :)

[stevenxxiu]

Well the tool has good intentions but it just isn't that well written. This makes me rather nervous in using it. The main problem I saw was that it treated strings and bytes the same, and lots of functions were called assuming they were the same thing. Files were opened in both 'w' and 'wb' and passed to functions as if they were the same thing.

The torrent files were parsed in a rather non-documented way, with the torrent file specifying its own encoding. I somehow got those tests to pass.

I agree there aren't any similar tools, but there are individual tools for their own file formats, like cksfv for cfv and md5sum, sha1sum for the hash files, aria2c for torrent files, etc. I'm gonna just use the individual tools.

But yes I can continue a bit more today see if I can make more progress. I don't intend on using the tool though.

[stevenxxiu]

Alright got the tests down to 26 failing ones. The remaining ones all appear to be due to a similar reason. I'm not quite sure what the test does however:

>>> failed test: stdin/out of -ZNVRMUI --unquote=no --fixpaths="" --strippaths=0 --showpaths=auto-relative --progress=no --announceurl=url --noprivate_torrent -tbsdmd5 -C -f- with file data4 
MD5 (data4) = 96d879de0782e286d4031a8de9e351f2
-: 1 files, 1 OK.  0.000 seconds, 19038674702.3K/s
- : undecodable filename
1 files, 0 OK, 1 file errors.  0.000 seconds, 0.0K/s
FAILED (3) ((0, 16))

Perhaps you could debug it with both Python 2 & 3 and see how it's failing?

I might stop working on it here on, since doubt I'm gonna use this in the future anyway.

[lxp]

Thanks for your hard work! I will look into it, when I have some free time.

[alfs]

Just a heads up - cfv is marked for autoremoval from testing on 2020-04-17
because of the python2 removal (https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=936288).

I guess we can re-add the package later on unless the current state of the py3 port is complete enough?

[lxp]

Thanks for this heads up!
Sadly, it is unrealistic that there will be a working py3 version until 2020-04-17.
However, my plan is still to release a Python 3 version.
Is there a way to extend the deadline and keep the py2 version a bit longer in Debian?