Notes: Work

The daily award for the most philosophical code goes to the author of this elegant way to check that two boolean variables are not equal to each other:

If DataStructure.Property("AmountVATIn")
    And ((DataStructure.AmountVATIn And NOT SearchPriceIncludesVAT)
    OR (NOT DataStructure.AmountVATIn And SearchPriceIncludesVAT)) Then    
    Price = RecalculateAmountOnVATFlagsChange(Price, DataStructure.AmountVATIn, TabSectionLine.VATRate);
EndIf;

I'm thinking of adding something like “And Not (DataStructure.AmountVATIn = SearchPriceIncludesVAT)” here to spice it up with a subtle note of insanity.

I like to notice things in the world around me that clearly have a story behind them. In video games, this is called “environmental storytelling”: they don’t tell you the story directly, but if you look around, you can guess which gun was hanging on the wall and who fired from it.

For example, I recently celebrated the company's 10th anniversary with colleagues at a local golf club. Balls had to be sent flying from the second floor; there are no railings for obvious reasons, but a net is stretched in case someone loses their balance.

Why are you talking about this, you ask? Well, there are warnings on the walls: jump into the net of your own free will and pay ten thousand dirhams. Recording this heroic leap of faith on camera is fine as well, just prepare five thousand more.

Do you feel the smell of a good history?

Another example: once flew to Turkey to rest and decided, just in case, to look through the rules of the airline (what can be taken on board, what can not). Among the list of items prohibited from being carried on board, I found “steel spear” and “steel flail” 😬

UPD: Another great example from somewhere on the Internet.

This week, I developed a REST service to set up our service manager (this is a configuration for managing a 1cFresh instance). Deploying the development environment is a regular task for us, and every time the manager's database had to be tuned by hand: tweaking the storefront, changing application addresses, overwriting scheduled tasks, and so on.

The implementation was simple. Come up with a JSON structure, write a parser, find a code in the configuration, make it work by external call, and make sure you don't break anything. Routine work, in general, but I love to do such things from time to time: I mean, to look around and try to figure out which of the daily tasks is annoying enough.

This one is a good example. To be frank, setting up the Service Manager wasn't a problem (launching the app and fiddling with the settings), but it was a thing to pay attention & spend time to. What do we have now:

1. There is a JSON file with all the settings;
2. There is a REST service for its processing;
3. There is a script that will put the first into the second;
4. There is a pipeline that will do it all by itself.

In short, a boot was rubbing a leg, and now it isn't. Yahoo!

The more I explore the Bitrix24 REST interface, the more I am amazed at how different its developers mindsets are. It is expressed in different ways.

Let's take, for example, the interface of deals and product rows related to them. There is no amount field in the table of the products: like, you need the amount for each line – count it yourself, that's it. However, there is the amount field at the document level! Can you guess what the field is named?

AMOUNT? DEAL_AMOUNT? DOCUMENT_AMOUNT? AMOUNT_TOTAL?

You didn't guess, the correct answer is OPPORTUNITY.

Where'd I digress? Yeah, a product line. It contains a product, a VAT rate, and a unit of measure. All three entities are completely independent: each has a separate table with auxiliary information and its own unique identifier. It is logical to assume that identifiers are stored in the product line: product ID, VAT rate ID, and unit ID.

Well, yes, but no. The product field actually contains ID, but for the VAT rate field it's a rate value. What's for the unit of measure field? Well, it contains a measurement code ¯\_(ツ)_/¯

Database normalization? What? What does it mean? Back off, man, you're distracting us from work.

For the last month I have been rewriting standard data exchange between FirstBit ERP and Bitrix for a client task. Co-workers doing the same on the Bitrix side prepared a huge mapping for this case: which field on the 1C side should be transferred to which Bitrix field (and vice versa).

They published this mapping as Google Docs tables, in the interface of which you can see users using any document at this moment — both logged in and anonymous. Anonymous ones traditionally are displaying as animals.

Colleagues generally prefer to work anonymously. As a result, I definetely used to feel like a Disney princess: you start working in the morning, and anonymous quokkas, penguins and chinchillas roll out from everywhere :-)

Someone told me about the Croatian comic strips NOT/BUT at the beginning of last year, when I was seriously learning to draw. Each strip there is about some kind of traumatic or simply gloomy thought that comes into the artist’s head while working. The idea is to push a reader in the right direction and give them a more practical perspective on the situation they're in.

I quit drawing a year later, at the end of February: it became clear that I no longer had time for a hobby. But comics are completely universal; look through, even if you have no idea what a kneaded eraser is :-)

All these endless soul-searching are well known to any professional, and the way out of them is not always obvious. Especially if you are angry, tired, and the deadline for the project expired somewhere last week.

Oh, shoot! Just watched the video and realized that I got used to writing OK in morning statuses when asked “how do you feel today”. Like, everything is normal with me, I'm alive and doing well, nothing affects my work, etc.

Now I'm afraid to guess what colleagues thought about me. I hope they also did not know that okay in the answer to such a question does not mean okay at all :-)

It is a bit late to sum up, isn't it? Well, the previous couple of months have been, um, tight, so I had no actual time to sit down and think. Now I'm on vacation for 2021 – so no time like the present. I'm writing, like, from the past.

Write out each achievement doesn't attract me at all. The year definitely turned out to be good: I did a lot of complex work on FirstBit ERP (wrote new modules, rewrote existing ones, crushed bugs, wrote functional and load tests – yes, there were countless things). This had a good effect both on the configuration itself and on the profits of my company. In addition, I passed 1C expert exam and a PostgreSQL professional exam. Hip-hip-hooray.

There were fails as well. I still feel lazy to jot them down, but the main one is obvious – I became monstrously fat and instead of losing ten kilograms (that was the original plan) – I gained five more.

Looking back, the root of most of the problems over the past year is this: I focused too much on task flow. They were all interesting in their own way, from a carefully written technical task to challenges like “find a problem in the code using your witcher's instincts”. You do one task, another, a third, a hundredth, and you do not notice that you have ignored almost everything except work. As a result, you get a lot of experience, but the world dries up to the Configurator window and there is no strength to somehow apply this experience. Unhealthy bullshit.

In short, if you imagine me as a character in some video game, it looks like this: attack and intelligence have grown noticeably over the year, but health, agility, charisma, and almost everything else in general, have dropped. Now I need to do something to deal with this mess.

You know what? I'll take ten kilometers run for a start.

Maybe, birdie, maybe.

I'll tell you about a funny and a little embarrassing case that I took apart in January. The essence in a nutshell: a huge auto-test based on Vanessa, which is intended to check the VAT calculation, falls somewhere near the end.

I start investigating. First, I look at the screenshots in Allure: OK, the reason is obvious – in one of the documents, the conditional appearance for the field with the VAT amount doesn't work. The test expected it to be unavailable if the VAT rate is zero, but it turned out to be available somehow.

It's a mess, it needs to be fixed! I look at the condition in the code: well, it locks the field if the VAT rate is in the list of “zero” rates (list of VAT rates whose rate is equal to zero). Everything appears to be simple and logical. What the hell could possibly go wrong here?

Well, I try to reproduce the bug manually. And there, all of a sudden, everything is nice: the conditional appearance works as it should. Floating bug, or what? I run the auto-test again, at the right moment I jump in with the debugger and find some outright garbage: in the list of “zero” rates, besides themselves, there are a bunch of empty links!

Frankly speaking, I was scratching the back of my head. The document receives this list from the common module with this code. An empty ref from here, even theoretically, cannot be obtained. Moreover, the module has the “Reuse Return Values” option ​​enabled, and the function is actually executes once somewhere at the beginning of the test, before all complex data manipulations. So, the test cannot affect it in any way, in theory.

Is it a dead end? Well, experienced colleagues have probably already guessed everything, but I had to dance around the bug and even check the standard, until got the following: return values ​​cache in 1C can be changed. I mean, not by calling RefreshReusableValues​​(), but by changing reusable values directly.

How? Well, if you get some values ​​from a cached common module, and they are not of a primitive type (string, number, etc.), you will not get the value itself, but a pointer to it somewhere in memory. You write this pointer to a variable and try to change it – you change the cache.

It's that simple, yes. This is what happened in my case: the form of another document called the method that generates the list of “zero” rates. Having received a list of values, it added an empty ref to it and used it in its logic. Thus, each time this form opens, the list cache contains more and more empty refs, which eventually broke the document at the other end of the configuration.

In a good way, the platform should throw exceptions when developer trying to change the cache, but until this happens, you have to take care yourself. For example, when developing cached modules, return immutable data types from them (FixedStructure instead of Structure, FixedArray instead of Array, and so on). True, this is not a 100% protection: firstly, fixed types are not applicable everywhere, and secondly, even in the latest versions of the SSL, this is far from being done everywhere. Do you know many configurations not based on SSL?

Sonar also know nothing about the problem, as well as less popular software. No silver bullet, in short – check your code, look at the code of your colleagues and try not to forget about another elegant way to bang ourselves in the foot.

A few days ago, there was a link to 1Ci certificate in the team chat (a colleague passed their junior course). I go over and see no auth form, no errors — I just get a PDF file. Everything is OK, right? Well, then I open the file:

Honestly, I am even delighted. We need to implement this UX in our products: you press, for example, the “print” button for a document, and it gives you a PDF! And inside it — a link to another PDF! And inside it — “entity is not filled”, so go fill.

You can sell it as a Russian way of dealing with errors. This is, like, for historical reasons! Blame all the damned matryoshkas, and Koschei with his needle. It is, as you remember, in the egg, and it is in the duck, which is in the rabbit, tucked into the chest, and further along the chain of nesting.

The nice episode of the “We Are Doomed” IT podcast about developer burnout. No insights, but you can hear something useful for yourself. I liked the analogy with video games, somewhere close to the middle:

There was such a game — Diablo. RPG, all sorts of spells, you know. A character has mana and health, and when you have no mana to cast… Ha, I sound like a nerd! Well, nevermind. Briefly speaking, when you have no mana to cast, it's taken from your health.

— Doctor Cat

I was developing payment documents in our configuration last week and came across an incredibly redundant solution to a primitive problem. Sorry, I can't keep to myself.

Just imagine: you have a document, which contains several tabular sections. Each of them has a comment field. You make a print form for this document; if at least one row of any TS contains a comment — you need to use one template 1, if there are no comments — template 2.

The task is pretty primitive, isn't it? All of us have done something like this a million times. Just read the selection, use IsBlankString() on the comment field and load the appropriate template. Coffee time!

However, instead of a short cycle, I saw this:

There is a chain of queries, in the lowest of which we scan all the TS (which, I remind you, we just raked out for printing). We are looking for comments in them, then we group the result several times and return it to the script.

Well, I'm not even talking about the load on the DBMS (I would venture to guess that this trick doesn't give a noticeable effect — after all, the selection is going to be small). It's just… Well… Checking a selection of rows is, like, five lines of code. Clear, simple, short, Sonar has no room to swear. How could you give birth to this? Because of great love for queries?

You know what? I bet that it's the correct answer. I can almost see this programmer, who has just mastered the query language more or less tolerably. He is in the absolute delight of new opportunities, so… If all you have is a hammer, everything around looks like nails.

This tweet reminded me of a job I once worked on. A client gets in touch: 1C is slow, do something. We ask a few questions. Okay, not all of 1C, just one operation.

They’re running a heavily rewritten Trade Management configuration, and the slow bit is custom code. Inside, it’s queries nested all over the place, a lot of messing about with value tables, some dubious calculations, and a bunch of server procedures seven hundred lines long each. We can’t copy the infobase, the technological log is unavailable, and DBMS and performance counter access are blocked. Debugging on the 1C server is off, and enabling it isn’t an option.

A mess. How can we find where it bogs down? Well, we can change the configuration; that’s something. I time the operation: yes, it thinks about something for a couple of minutes. Excellent! Rubbing my hands, I narrow the search by adding a whole pile of measurements to the most suspicious sections.

And here’s the funny part: nothing’s slow! The code doesn’t exactly fly, but we’re nowhere near minutes. For something this complicated, the timings look fine. Could it be serialization between client and server? I go through the server procedure carefully. Nope, nothing’s being passed back. Everything on the server stays there.

I move to the client code. What could possibly be slow here? Simplified, it looked something like this:

I stared at that code like an idiot for about ten minutes. Eventually, I asked a colleague, who told me something surprising: Notify() doesn’t just send notifications to every open form, it waits for their notification handlers to finish.

That blew my mind. I went digging and eventually found a form that caught the operation-completed event and immediately refreshed its dynamic lists. A query in one of those lists caused the delay: a subquery inside a subquery inside a subquery, plus a pile of joins “through a dot.” Nothing interesting after that.

Well, there you have it :-) Notify with care!

On the screenshot above my little one is just letting me know where she is: walking down the street, at the school, at a playground. It's not her fault that I have a sort of nerd view: when I look at this chat I see nothing but code on Gherkin, which is just a bit broken. A little patch here and there and it will be fine :-)

And I exit home
Then I enter school
And I exit school
Then I gonna swing

We use Gherkin to write our autotests for Vanessa Automation. I can't say I've made very many of them, especially in comparison with some of my colleagues… Well, it seems like I've made them enough to twist the reality a few.

For posterity, I’ve uploaded one of the inefficient queries I worked on a few months ago to GitHub Gist. It powered a dynamic list on a catalog item form, and whenever the user opened that tab, the platform went off to meditate, even with a relatively small database.

At first glance, it looks straightforward: a bunch of queries against document tables, joined with UNION ALL. Each table is filtered by roughly the same conditions: reference type, date, and whether the reference appears in a subquery’s result.

But there are several potential problems. First, at least 14 tables are involved, one per combined query. That alone raises the risk that the optimizer won’t have time to choose even a reasonably decent plan. Most likely, it will just take a wild guess and hope for the best.

Second, much of that work is probably wasted. Each query has something like this:

WHERE VALUETYPE(AdditionalExpenses.Ref) IN (&DocumentsListSelectedTypes)

The dynamic list receives a list of document types to display. But that filter kicks in after the data has been read: even if the user only wants invoices, the DBMS still reads all 14 tables, then discards 13 of them.

Still, none of that is especially critical. If that were the full list of problems, we might never have investigated. The main issue is the second condition in the WHERE clauses: each query checks whether the document reference appears in a subquery’s result.

Using subqueries in conditions is almost taboo unless temporary tables are involved: the DBMS often can’t work out how much data it’ll get back, or how best to handle it. And the data source makes things even worse here. A filter criterion isn’t a database table you can read, even by scanning; it’s a collection of tables. DocumentsByProject contains thirty!

That sounds bad enough, but there’s more: each of the fourteen combined queries selects documents of a particular type from those thirty tables.

And… scene :-)

We recently swapped out SSLi’s creaky old homegrown versioning for the platform’s built-in data history in our configuration. I’m now finishing data history backup and restore. Surprisingly, neither the Standard Subsystems Library, the Standard Technologies Library, nor SSLi can do this yet, though I didn’t expect it from the last one.

I’ll write more when it’s finished. For now, here’s a neat trick that came in handy: the platform’s built-in data processors available through “All Functions” can be exported as ordinary epf files! Colleagues on Infostart explained the trick in detail here and here. In brief, the magic looks like this:

КопироватьФайл(
    "v8res://mngbase/StandardDataChangeHistory.epf",
    "Q:/StandardDataChangeHistory.epf"
);

The links above have the full list of forms and processors you can pull out, how to get that list, and plenty of people arguing about it all.

Why might this help you? Honestly, I don’t know. In our case, we were building interfaces for data history and were curious about how 1C had written its own. Spoiler: it’s pretty messy.

I want to share a couple of great pieces about modern software. Neither is exactly new, especially the first, but I’m sure some of you haven’t seen them:

• Software Disenchantment by Nikita Tonsky
• Everything in Software Is Amazing, but Nobody Is Happy by Evgeny Trifonov

What brought this to mind? Last week, I took part in a hackathon for developers at our company. I learned a lot of cool things and even shared first place with two other people who came up with very similar solutions.

(If you also work at First BIT, the results are on the portal, along with a review by another winner, Dima Leshchenko.)

Along the way, we had to deploy and configure a mountain of software: EDT, RabbitMQ, Docker, GitLab, JIRA, SonarQube, and a wagonload of smaller, simpler tools. All right, no complaints about RabbitMQ: it’s light and fast. But the rest… I already knew EDT had an appetite, but GitLab and JIRA’s hunger really surprised me.

Yes, everything ran in Docker; yes, the configuration wasn’t optimal, with several PostgreSQL servers instead of one, for example; yes, it was Docker for Windows, which every sysadmin I know loves to make jokes about. But twelve gigabytes of RAM straight out of the gate?! I won’t even mention the CPU: the load looked as if the computer were simulating a nuclear explosion in real time.

It’s hard to find a better illustration of Nikita’s points: this software devoured a sea of resources and calmly asked for more before it had even started doing anything useful. But Evgeny has a point too: with one single command, I got several operating systems up and running, complete with some pretty complicated servers. And it took a tiny fraction of the time it would have a few years ago.

I don’t have an answer. I don’t think anyone does. We’ve been having this argument forever. Pick either side, and you still end up trying to find a balance. You need to write good, fast code, while also avoiding premature optimization. Getting the code right feels great: you remember that, damn it, you’re pretty good! The second simply keeps you from losing your mind in search of the perfect bubble sort algorithm :-)

I’ve put a data processor on GitHub for setting up the data history feature that arrived in the platform not long ago. Here’s what it does:

1. Builds a tree of objects that support data history;
2. Shows which objects currently have data history enabled;
3. Lets you enable or disable data history for objects and their attributes.

It supports batch operations: for example, you can enable history for several documents and all catalogs at once. Or wipe all your changes and go back to the defaults the configuration developer set.

I think it turned out well. I mostly wrote it for myself, to give my brain a workout in the evenings, but we might even include it in our configuration! Right now, SSLi handles our data versioning, and it’s not exactly efficient: it’s slow and makes the database balloon. Hopefully, the platform will do better when we test it.

You can find similar tools on Infostart and ITS, but they’re either buggy, work only with Russian configurations (no English interface and dependencies on the Russian Standard Subsystems Library), or are just out of date, with no support for constants, for example.

The architect of our main product (who also happens to be my team lead) talks about how we implemented 1C in Dubai. He doesn’t get bogged down in technical details, so you only need a rough idea of the subject to follow along.

The video was recorded at a developer conference in Krasnodar last June, so a few things have changed since then. For example, we now have working prototypes of the application in Arabic, and our automated testing has grown so much that Miss Vanessa pretty much deserves her own spot on the team :-)

Here’s the brief: a client’s month-end closing document takes just under an hour to post. The database isn’t especially big, but an hour is still way too long. Sure, you don’t close the month very often, and you could do it at night. Still.

I dig into the code, profile it, and see that the platform spends almost all that time in a single procedure, executing a query batch. I look at the first query and think: classic, accessing data through a dot on a composite type. The DBMS must be attaching a wagonload of joins to huge document tables, and the optimizer can’t come up with a decent plan in time.

I test the theory. But SalesDocument includes only eight document types. That should be manageable: the usual rule of thumb is that the optimizer can find a decent plan with up to eight joins.

I check the document tables: they aren’t particularly big. I run the query separately from the batch. It isn’t instantaneous, reading around 350,000 records and selecting about 200,000, but it certainly doesn’t take an hour.

All right, the batch’s first query is heavy, but it’s not the problem. I look at the second and realize that we’ve just read the entire Inventory register and selected most of its records, and now we’re reading it again and joining both result sets on a pile of conditions. The register’s record table has no suitable index: only the standard recorder index, which naturally doesn’t help.

I check: this is exactly where the platform spends most of its time waiting for the DBMS.

I couldn’t find a way around reading the whole register table twice. Every alternative I tried meant changing how the data was stored, and that wasn’t an option. So I went with a compromise: in the second query, I replaced the join to the actual register record table with a join to a temporary table created in advance and indexed on the join fields.

This was a case where 1C’s general recommendation worked perfectly: posting time fell from 50 minutes to three, and after further code optimization, to one minute. Fifty times faster than where I started.

A minute for month-end closing is usually fine, so I called it a day.