Open source development hits another roadblock

Open source lets anybody contribute to any code but it does nothing to ensure the contributor understands either the code they are contributing to or the code they contribute. The comment in front of my is an example. The comment is a reply to another comment. The first one suggests approach A to fix a problem. The reply says approach A is bad because it will create problems in the future. The reply then suggests approach B. Approach B also creates problems in the future plus approach B is a dangerous unreliable approach. The software us web related and approach B can break a Web site if not executed perfectly. Many, perhaps most, Web site owners will not have the access to complete approach B. Approach B can also change data outside the target area and have unexpected consequences almost impossible to test without visiting every page in the whole Web site with an automated A/B comparison.

So approach B is dangerous, unreliable, and a real risk of having unforeseen consequences. How can it be preferred over approach A? The person suggesting B probably has very little experience with the relevant software. The person suggesting B is most likely a developer who has never had to maintain an operational site. The person suggesting B may not understand what it is like to own a Web site and depend on contract developers to not install things that are impossible to understand.

Your approach has to be reliable and not break anything. Your approach has to be predictable, do exactly what is expected, and nothing more. Your approach has to be easy to trace backwards after you leave the project. There is rarely anything like this taught at school. If there was, software would be as easy to use and reliable as washing machines (excluding the stupid European and Korean models sold in Australia but not designed for use in Australia).

Ok, so washing machine manufacturers have problems designing their machines to wash clothes. They suffer under the abuse of marketing people who want the machine to do everything else even if it makes the washing machine useless for washing clothes. Open source developers do not have that excuse. Their reason for doing something one way is to satisfy their own requirement at the time of development. That does not mean they should jump on and suppress a different approach.

Some open source projects add automated testing of code execution or code format. Testing the format of code is usually the first step and accomplishes nothing, given that the standard enforced is usually a bad Unix format never adopted for any reason other than habit. If open source was truly open, we would see progress on code readability., we would see semantic code standards.

Automated testing of code execution is often only as good as the code development. Using the Drupal code testing as an example, the test is often developed by the code developer and the tests are often only their to prove the code works under ideal circumstances. The ideal circumstances may not be documented to a point where users of the code can comply with the ideal.

The Drupal core code still crashes due to faulty code in user modules. When you trace the data flowing through the code, the user module is often passing valid data to a function that then passes the data on. After several passes of the data from function to function, the data hits a function that cannot handle the data. If you report the error to the Drupal developers, the report is usually dismissed until a large crowd of dissatisfied users all complain about exactly the same thing in the same issue. An argument may then start about which one of the several functions should check before passing on the data.

Some of the testing I pioneered in the early days of the Web is really easy in PHP and Drupal makes it an absolute breeze. if the best of the PHP and Drupal features were applied to both test code execution and create the documentation, many of these core problems would disappear.

Good design

There are no tests for good design. Good design is something I used to teach back when I taught application development. If you want to create a test for good design, you define the testing before you start writing code. The code is then created to satisfy demanding tests.

Almost all open source contributions are developed to fit only a single requirement for one person at one site. If the code passes that test, the developer stops testing. You then get enhancements added by other opel who have different requirements and they test only their part against their requirements. Rarely is there a combined overall exhaustive test of the combined code.

Applicability

There are no tests for applicability. Applicability is one of the reasons Drupal has 15000 add-on modules. There is no way to test the applicability of the 15000 add-on modules when so many have overlapping application. There is already a huge backlog of applicability testing to perform on Drupal just for the database access code. The switch from SQL to automated generation of SQL leads to SQL that is not the best choice in all cases. The bad cases can be extremely disruptive and damage the performance of your Web site. The people with the loudest voices in the Drupal community then recommend approaches that suppress the symptoms instead of fixing the problems.

If a project as big and as successful as Drupal cannot fix some of the basics, what do you expect from the rest of the open source community?

Accumulated problems

Drupal, as an example, suffers from the same problems that destroyed some mainframe projects despite hundred million dollar budgets. Drupal leaves too long between releases and puts too many changes in each release. A move to a more agile responsive development process would help.

Drupal also incorporates some changes based on fashion. If you are in the Drupal community, are pushy, and good at politics, you get your changes promoted over more valuable and useful changes. Some really good ideas are squashed by negative comments from people who pop up out of nowhere. There is no requirement for the detractor to prove their case, they can just say anything no matter how irrelevant or wrong. The contributors stop contributing. The space remaining is then filled by something flashy sounding that may turn out to be unproductive.

Some of the results are a pile of band-aids. Each one hiding the symptoms of the problem underneath. There may eventually be documentation but the first round of documentation may be a year after the code is in place and may not describe how you use the relevant feature. When you try to follow the documentation, the examples in the documentation might work if everything else in the system is exactly the same as the system used by the person who wrote the documentation, or they might not.

Using Drupal as the example, and knowing that Drupal was based on functions calling functions, but now is an inconsistent mix of functions and objects, I sometimes trace Drupal crashes through their code then through their documentation. I may contribute back both improved code and improved documentation.

The code trace may show 11 or more functions between the code supplying data and the code that crashes. None of the functions have the responsibility of checking the data. The improved code, to insert a check somewhere, is usually rejected without a valid reason by all 11 function maintainers. The improved documentation is difficult because the code related documentation is not as open as the other documentation and none of the documentation is allowed to prove anything.

The larger projects are crashing into problems created from accumulated contributions that misdirect people and the projects.

Divide and conquer

Mambo divided into Mambo and Joomla. Joomla survived and expanded. There are times when a new direction works.

Using Drupal as an example, Drupal 7 abandoned the small Web site for the large corporate market while proclaiming Drupal is the best choice for small sites. Drupal as one project is caught in the middle. Dividing would help Drupal develop one version for the corporate market and a simpler more efficient version for everyone else. Just the simple act of separating some of the bullies from people who want to use finished Web sites would help.

Another example is the Openoffice/LibreOffice split. At this stage there is no visible difference. LibreOffice was a little bit behind OpenOffice then progressed faster. They are at a point where they are totally compatible and ready to be different. What will they do? LibreOffice split from OpenOffice when OpenOffice was under the control of Oracle and Oracle where likely to force more of the OpenOffice development toward Java. The split could free LibreOffice from Java, giving them a clear advantage. The split made existing developers enthusiastic and brought in new developers. Now they need a real point of different to aim for.

Unfortunately we cannot see a difference today and office applications are too important to bet on rumours. I had to switch some computers from OpenOffice to Microsoft Office because of minor errors in both OpenOffice and LibreOffice. If the lost time was used to fix OpenOffice instead of splitting OpenOffice, those computers might be running OpenOffice. The refreshed LibreOffice might leap ahead to prevent more defections but it will be a long time before people start to return; unless LibreOffice announce a solid significant change in direction. Throwing out Java would be a significant change but not something of meaning to most office application users. Perhaps LibreOffice could add a multimedia feature, something like ACDSee image management, to push their product out in front of both OpenOffice and Microsoft Office.

Conclusion

The larger open source development projects have outgrown their development approach and devolved to the same back room politics approach limiting commercial software. The one long term survival trait is the frequent branching of open source projects, renewing support for the projects, and giving them the chance to bypass the roadblocks.