Notes: Python

Released a new version of my console timer for intermittent fasting. I wrote this app about a year ago, when I was once again upset by the Zero application for Android: some very primitive functions (like viewing a specific interval) did not work. Oh my gosh, guys, you had one job!

The main difference between 1.3.1 and 1.2.3 is that the console menu has been cut to hell in favor of the good old arguments and options. The menu concept looks convenient only if an application has few features. However, as soon as you take a step forward, the need to answer a list of questions each time you need something starts to irritate you.

The other day, I was working with cookies in one of my scripts. While searching for the optimal solution, I came across an absolutely charming (100% working, by the way) tip from StackOverflow:

You can get a CookieJar object from the session with session.cookies, and use pickle to store it to a file.

So, literally: keep your cookies in a jar, and to store them, pickle them. The jar with pickled cookies, by the way, can be put on the shelf later.

How can you not love python after that, huh?

P.S. While writing this post, I got curious – why pickle and not serialization? So, briefly: this is the way.

I’ve published a Python script for finding long-running queries in the 1C technological log. I wrote it in a fit of despair: I couldn’t understand why my trusty Bash script reported an average execution time higher than the maximum for one query.

The problem turned out to be gawk. With some log events, it couldn’t work out the duration: it tried to convert a string to a number, failed, and… Oh, no! Of course it didn’t throw an exception! It simply treated those strings as zero and calmly carried on.

The fix was even sillier than the problem: I simply added an explicit string-to-number conversion, and everything worked. How the hell is an explicit conversion different from an implicit one in gawk-land? And, more importantly, why?

Well, that was a circus. At least I got a new script out of it, and it goes easier on memory and CPU too. In the Bash version, I used three collections to calculate execution count, total execution time, and maximum execution time, each keyed by query text and context. All three had to be updated and kept in memory.

That works until you get into hundreds of thousands of entries. Then the collections start eating gigabytes of RAM, and searching them burns through CPU time. The new script is more straightforward: one collection stores all the data for each query.

In my previous note, I discussed tasks where Bash is the perfect tool: minimal constraints and simple requirements. I also mentioned that sometimes it can’t cope at all.

Here’s one I actually ran into. Suppose we have around seventy gigabytes of 1C technological logs and need to rank exception descriptions from most to least frequent.

Easy, right? Extract EXCP, get Descr, count repetitions. I’d even written the necessary script before. I run it and patiently wait…

Know how long it took? Neither do I: after a day, I stopped it and investigated. Something was getting stuck inside gawk: it was hammering the CPU, with cat occasionally chipping in to read more data.

Well, crap. I thought about it and rewrote the script in Python. The new version finished in fifteen minutes and gave me:

1. A ranking of exception descriptions;
2. A script I can read six months later without Google;
3. Confidence that I can add two or three conditions without accidentally summoning Satan through a combination of operators and flags.

Of course, I could have optimized the Bash version. The array lookups are probably what’s slow: the more descriptions you’ve collected, the longer each search takes, growing linearly with the array size. I could change the collection approach, using gawk only to extract descriptions and passing the results to sort and uniq.

But that starts to resemble the meme about a loaf of bread: you can build a trolleybus out of it, sure, but why? Come on, I just need to solve the problem. Python already did that with flying colors, saving me a lot of time and nerves.

That’s my point. We know Bash has its quirks, and often we can live with them. We know what it’s good at too. But sometimes the question becomes: is getting a solution specifically in Bash worth spending two hours wrestling with utilities, parameters, manuals, and Stack Overflow posts?

That’s why not Bash.

UPDATE: A good post by Ivan Grishaev on the same subject.

I occasionally see colleagues asking: is 1C still messing around with Bash? Why not Python? Or PowerShell, at least? Weirdos!

Yes, Bash isn’t native to Windows, which is native territory for 1C, and getting it onto Windows is a job in itself. Yes, as a task grows more complicated, script readability drops exponentially. Yes, Bash simply can’t handle some tasks.

But its main advantage is that, for many 1C technological log analysis tasks, it is the simplest and fastest solution. Like an old screwdriver your grandfather used: always at hand, scratched, nicked, and a little awkward, but still working perfectly.

An example? Suppose we need to extract exceptions from the log. Let’s keep it simple and say we only need the first line of each event. We get something like this in Python.

You’ll say it can easily be cut in half, or even to a third. After all, it’s just a one-off job. Why fuss about readability and predictability?

Agreed! But let’s first solve the same task in Bash:

grep -r --include "*.log" ',EXCP,' > result.txt

That’s why Bash.

I posted on GitHub a Python script which I use to sync files between my PC and a NAS. I have a Synology DS220j; a lot of software comes with it, including a backup utility. However, it seems to have been made for show only: the program began to fail even at the setup stage, so I lost confidence in it pretty fast.

I suffered with the problem for some time and eventually returned to the good old rclone, which has only two problems for me. Firstly, I can't normally connect to an SMB share: yes, the login and password can be saved in Windows and rclone will use them, but OS will forget it as soon as possible. I got out of the situation by connecting my NAS as an external drive.

Secondly, I had many files and folders to sync: a directory here, a directory there, a config from there, a profile from here… In order not to produce spaghetti code, I wrote a simple script that takes sources and receivers from a config file, and then executes rclone for each pair.