I moved this blog to my professional site.

All the posts available in this blog are now available also at http://www.toscalix.com  Every new post will be published there only.

Why Codethink is a founding member of the Civil Infrastructure Platform, a Linux Foundation initiative

This blogpost was originally published on the Codethink website on Thursday March 9th.

On April 4th 2016 a new Linux Foundation initiative called the Civil Infrastructure Platform was announced. CIP aims to share efforts around building a Linux-based commodity platform for industrial grade products that need to be maintained for anything between 25 and 50 years - in some cases even longer. Codethink is one of the founding members.

Industrial grade use cases

In order to describe why this initiative is relevant let me go over the use cases that motivate companies like Siemens, Toshiba, Hitachi, and Renesas to share efforts.

During the Open Source Leadership Summit, Noriaki Fukuyasu (Linux Foundation) and myself, based on the experience of Siemens, Hitachi and Toshiba, described the development life cycle in industrial grade use cases. For example, a railway management system is as follows:

• Analysis + design + development: 3 - 6 years
• Customizations and extensions: 2 - 4 years
• The certification process and other authorizations take a year.
• Each new release or update has to go through further certifications and authorizations that take between 3 and 6 months.
• The system is expected to work for between 25 and 50 years.

So on average, an industrial grade product might take 5 to 7 years from conception to deployment. This is coherent with our experience in other industries like automotive, where life cycles are also quite long despite the expected lifetime being shorter.

A key part of the life cycle is maintenance. Due to its length, the associated risks are high. The certification processes to introduce significant changes in any already deployed systems are painful and expensive. In addition, the capacity to simulate a production environment is, in general, limited. This is true in other cases like energy production plans, for instance.

Open Source principles in the Civil Infrastructure industry

It’s obvious that Open Source could have a dramatic impact in this industry. By sharing efforts, corporations can commoditise a significant portion of the base system focusing on differentiation factors, increasing control through transparency and the quality of that starting point over time. Collaboration with upstream will bring even higher impact benefits.

Two immediate challenges come to mind when thinking about Open Source in this industry:

• Development of processes and practices to produce software for safety critical environments.
• Bridging the gap between the Open Source approach for software maintenance and the approach currently taken when building large-scale platform projects. For instance, how can approaches oriented to update any specific Open Source software component to the latest upstream stable version be compatible with any typical industry SDLC?

Can you reduce the gap?

We have for years been working on transformation projects for which one of the goals has been to reduce the gap between the software our customers ship and what upstream is continuously releasing. One of the key steps is to adapt an organisation’s processes using FOSS tools. Over the years we have been a strong advocate that the closer to upstream you are, the more benefits you reap from the Open Source development model, maintenance cost reductions being one of the main advantages.

So why did we get involved in an initiative that aims to maintain a kernel for 25 years then?

The short answer would be... because we love a challenge!

Safety critical with Linux-based systems is a challenge currently being faced in the automotive industry for instance, where Codethink is a strong player. When we analysed some of the industrial-grade use cases, it called our attention not just to the magnitude of the second challenge enumerated above, related with super long term maintenance, but also the apparent conflict between the industry requirements and the referred well known Open Source practices.

Hence the main driver for an Open Source consultancy like Codethink in participating in an initiative like CIP is to learn by doing, that is, putting the Open Source development, delivery and maintenance best practices under stress in one of the toughest environments. We bring our experience in producing embedded Linux based systems and our Open Source culture, to work together with industry leaders in finding solutions to these challenges, by looking at them with FOSS eyes.

Current activities

Codethink is participating in CIP in several capacities, the most relevant being:

• Kernel maintenance
• Testing tooling

Kernel maintenance

The first CIP approved kernel is 4.4, an LTS kernel supported until Feb 2018. Ben Hutchings is the initial CIP kernel maintainer. Besides providing support for the reference platforms, Ben is working on several activities like backporting the security patches, such as those from the KSPP and consolidating the maintenance policies, taking those from the kernel community as reference.

Testing tooling

kernelci.org is the most successful testing project in Open Source. Its impact in the kernel community is growing, as is the number of people and companies involved. It was designed and developed as a service where the testing activities can take place in distributed board farms (labs).

Codethink has been working on making the tools easy to deploy on developer machines through a VM, so they can test kernels on directly connected boards. This first milestone of the CIP testing project is called Board At Desk - Single Developer. This activity was described at the Open Source Leadership Summit 2017 and the first beta released during ELC 2017.

Conclusion

The challenges for Open Source that Industrial-grade product development and maintenance introduce are great, especially in two aspects: safety-critical and maintenance. Codethink is working on CIP to help the industry to overcome these challenges by adding our Open Source perspective.

Learn more about the CIP project by checking the following slides and videos from the conferences in which CIP members have participated.

Automotive supply chain and Open Source: a personal view

Software in automotive yesterday

The automotive industry has treated software like any other component, as part of the traditional, well structured and highly controlled supply chain. Tier-1's has been providing software to car auto-makers for some years now and both together have done what they have could to prevent consumers or downstream players in the supply chain from manipulating it, improving it, customising it nor from adapting it. It didn't matter if the software was Open Source or not, they have treated it as if it was proprietary, promoting locked-in practices. Only very few stakeholders with the right kind of agreement could manipulate it in a very limited way. Consumers and third parties didn't have a say.

For a software company, there has been no life outside the traditional supply chain of any auto-maker.

From all the reasons I've heard since I am involved in automotive, that justifies this situation, arguments related with security together with the safety critical nature of some systems shipped in a car are among the most popular ones.

But wait, once I buy a car, I can manipulate or even change the engine, the suspensions, the tiers, the brakes... but I can't even dream about changing the software? Eveny if it is Open Source?

Software in automotive tomorrow

In my opinion the current way software is treated by automakers, the supply chain associated to them and the current business model around software in automotive will change dramatically. And the main reason will not be the license of the software used but the fact that the increasing amount and complexity of the software shipped with any car, together with the challenges that connectivity and privacy bring, will open up the door for new new players with new business models, Open Source business models. Those new factors will also provide current stakeholders in the supply chain the opportunity to increase their services around software. New players will challenge the current controlled and centralised environment.

I believe the software supply chain will expand beyond the purchase of the car, providing inputs at every level. Initially this will take place in a semi-controlled way, specialy by dealers, but later on software will become a major point at every stage, where there will be a wide offering of security and performance improvements, deployment of new functionality, maintenance services, customizations, integration with third party services, etc., to consumers directly by ISVs, enhancing the user experience adding more choice and adaptability to their particular use cases. The role software companies today play as providers will be substitued by a partnership relation.

In the same way that we all have a relative that "fix cars", there are many developers out there that can put their hands on the provided software of their own vehicles. There are also many companies willing to do the same for their own fleet or somebody else's one. A new car cost a hundred times more than a mobile and its life cycle is at least five times longer. I do not think the mobile industry will be the mirror for autmotive.

I believe that the picture will be closer to the current one in the enterprise industry, even if the journey to get there is diferent. Some automakers might remain as key stakeholders when talking about software, but not like today. 

The consumers demand for more and better services and the portability needs across the different platforms and devices (cars) in order to make business with software at scale, will increase the pressure on OEMs and Tier-1s over hardware and software standarization. That pressure might become in some cases as strong as govements regulations, I think. It will be definetly stronger that in the mobile industry. I see this as a positive factor for consumers.

Experience shows that, if you intend to control a software ecosystem, you need to become upstream and create around your software a business model that serves as a service platform for third parties (ISVs). Maybe not even then you will be able to stay in control of what the customer is consuming. In other words, to be like Apple you need to control the hardware, develop at least the software platform and create a field for third parties to make money through "your platform", which means at least controlling the distribution and updates. 

I do not believe automakers will be able to achieve that same level of control being only a Open Source consumers in a connected world, by restricting access to your platform to anybody but those who are part of your supply chain. not even if they become Open Source contributors. And no, the Android model does not apply to a single hardware (car) vendor.

I am obviously no guru so take all this as nothing but a personal opinion from an Open Source geek. But if you think it is feasable for an automaker to achive similar levels of control with Open Source based platforms than Google or Apple has over their ecosystems in the mobile industry, I think you are at least as crazy as you claim I am.

Codethink is hiring engineers.

Codethink is steadily growing for some time now. I have been working for 18 months in the company and it is about time for me to tell you that this is an outstanding place to work if you:

• Love Open Source and work upstream.
• Prefer to work in a startup kind of environment than in a corporate one.
• Love to solve complex problem for first class customers instead of living peacefully in your comfort zone.
• Like to travel to customers and conferences once in a while.
• Understand that being challenged every day is the best way to improve.
• Like international and multicultural environments with a British touch.
• Think that Manchester, UK, is not a bad place to live or to visit once in a while. Hey, I am from the Canaries and too old to party night and day when I am not working, ut that might not be your case ;-)
• Like IRC, git, vim/emacs, RSS and some other technologies that newcomers to Open Source consider... old school.
• Are not scared of sending patches by mail.

But above all, you are willing to learn no matter how senior you are!

Codethink is an independent consultancy company based in Manchester, UK, although it has some people like myself working remotely. Our customers are also spread around the world. We are around 75-80 people now, most of them, engineers.

Most of our work is related with Linux kernel, low level system activities, compilers, delivery of linux systems, distro and SoC work.

Let me know if you could be interested in working at Codethink or send directly your CV to jobs@codethink.co.uk

Business model as a variable to consider when choosing Open Source software.

A couple of weeks ago I gave a talk at an Open Source event for professionals. A senior developer asked me a question that got my attention. It was something like...

So do you recommend to consider the business model when choosing a particular piece of Open Source software, beyond the license?

My answered was something like...

In general yes, specially when that piece of software is core to your product/business and definitely in cases where you distribute/sell it to customers.

 

Thinking about it later and reviewing some of my talks and posts, I thought it would be interesting to develop that answer a little further.

Key questions

Any analytic report about who writes the code in open and collaborative environments will reflect how corporations involvement is increasing in Open Source software development at every level. More and more companies are transitioning from becoming FLOSS consumers to producers and almost every new software company out there has Open Source as a core strategy or even as part of their DNA.

But who is sustaining the development of that key piece of software that will be a core part of your future product? Who pays those developers? Why? How does the key stakeholders benefit from the outcome of the ecosystem and the software they produce? How much do they invest in the production of that software? For how long? How do they get their income? What is the relevance of the software produced by the ecosystem they feed in their business models?

These and similar basic questions need to be fully understood before a specific software becomes part of your key product or business. Knowing the answers to the above questions might not prevent you from surprises in the future but at least can prepare you for the potential consequences. What it is clear to me is that these answers are becoming more complicated to find and understand over time, specially for those companies who do not have a strong background on Open Source.

Choosing a specific piece of software based on purely technical variables or even present healthiness of the community around the project/organization, expectations of the number of contributors or impact in general might not be enough any more. A specific community or project will become "your provider" so the business model behind it is equally important.

Examples

You might base your new product on a technology produced by a company that is heavily investing on it but that has no clear associated business model. It might be for instance a service company and the software might not be core to their services since it has no significant impact in their income. This might not be your case so you can end up relaying on a provider that do not have "your use case" as a critical one, no matter how big you are and how profitable your business could be.

Think about the case when your product relies on a technology that is disruptive but sustained by a start-up or a company that is not profitable yet. Engineers might be extremely excited about this new technology for good reasons, the community around it might be on fire at the point when you are analyzing it, but the future of the business might be unclear due to, for example, the inexperience of the executives leading the project, the market they are playing on or the strategy of the investors supporting it.

A well established project has suffered a fork, attracting the core developers. Very soon a boost of energy is noticeable in the development front, which leads to an increase in the number of contributors. They gain customers rapidly but it is unclear if they have a solid business case to sustain the development effort with the current growth in the short term.

The above three cases and many others out there show that, although there is no question that published Open Source software development represent an advantage compared to proprietary when it comes to sustainability, that doesn't mean it is at no cost. This is specially true when safety critical and/or long term support are essential variables for you. 

Sometimes choosing a technology/project with a business model more aligned to yours could work better than choosing the leading or hot technology/project. In other cases, going for a more conservative but solid/veteran project might be the right choice. In some others, assuming risks can be the best choice.

Key considerations

I believe that for any organization, moving from being a good Open Source citizen to becoming an Open Source company also means that:

• You evaluate the business models around the organizations/projects that produce the software you contribute to and/or consume. This requires involvement of business related professionals in key decisions related with selecting Open Source software.
• A key part of your sustainability effort is to create feedback loops with your providers also at a business level, contributing to their sustainability as Open Source project but also as organization, specially when you are bigger/stronger than they are. Ideally, your business model should be compatible with those from your providers so your success has a positive impact on theirs.
• You put in place mitigation actions to reduce the risk of severe changes in your key Open Source software providers situation that in the worst scenario, might compromise the future of your product or even company.

Conclusions

The fact that Open Source is done in a collaborative way might reduce sustainability risks in most cases compared to proprietary business models. But the nature and complexity (dependencies) of many Open Source ecosystems introduces new risks that requires to consider variables beyond the technical or community aspects.

This is why it is so important to choose carefully which FLOSS software you will use, produce and contribute to, looking at the business models supporting the selected project/technology stakeholders.

As a company, you will not control any more the relation with your FOSS providers as you used to. Open Source has changed the nature of that relation, that goes beyond the license or any further contract. To take advantage of this new reality, you will need to adapt.

Final reminder

Open Source is also about providing more control to creators. If you want keep similar levels of control that you had in the proprietary world, contracts will not be enough anymore. You will need to write the code yourself, which means, become upstream.

And the smartest way to achieve sustainability writing code is through collaboration.

Automotive, what an opportunity for KDE!

During the last year I've focused a significant part of my effort on driving the GENIVI Development Platform, together with some Codethink colleagues and other GENIVI professionals and community members.

What is GENIVI Development Platform?

GENIVI Development Platform (GDP) is a project and an outcome.

As a project, it can be defined as the delivery side of the GENIVI Alliance. Today is a fairly standard Open Source project, done in the open following many of the most common practices any FLOSS developer would expect.

As an outcome, GDP is a Linux base distribution (Poky) derivative, built with Yocto, that integrates the software that GENIVI community (automotive professionals) develop as Open Source software.

It still a small project but the quality of the platform and the number of people involved has grown this last year significantly.

GENIVI Alliance is a consortium of +140 companies so obviously most of the overall effort is done by paid developers. Changhyeok Bae (community member) together with Tom Pollard and Robert Marshall (Jonathan Maw before him) from Codethink Ltd, constitute the maintainers team, who are responsible for the integration, testing and release of GDP.

These guys are supported by people like myself, doing coordination, marketing, documentation, IT services, infrastructure, testing and many other key tasks which provides the project a level of robustness and scalability that any serious attempt of this nature requires nowadays.

I am interested, where can I get more?

You can find more general information about the project in the following resources:

• GDP Project wiki
• Communication channels:
• Mailing list
• IRC channel: #automotive in irc.freenode.net
• Slides and videos from talks about the project.

GDP-ivi9 is the current stable version although we have moved so fast this last year in terms of the software shipped, that I recommend you to try the following:

• If you are interested in a solid base system, try GDP 11 RC2
• If you are interested in checking the latest UI and demo applications, try GDP 11 RC3
• If you are interested in building from scratch you own images with the latest software, do it directly to GENIVI's rolling release, called Master for now. You can get the latest software there. It should most ly work since we put stabilization effort on it, following the openSUSE Tumbleweed mindset in this regard.

Currently GDP supports RPi2 & 3, Minnowboard MAX and Turbot, Renesas Porter and Silk and Qualcomm Dragonboard 410c. GDP ships Qt 5.6 at the moment, since it is based in Yocto 2.1...

...which makes GDP a great target for KDE software, specially for Plasma.

GDP and KDE

Putting the effort on having KDE well supported in Yocto would provide the project a third life, landing on an industry that is heavily investing in Open Source with a key piece of software, with no clear competitor today in the open.

It would revamp the interest of many KDE developers in porting their apps to embedded/mobile environments and would bring attention to the project from Qt professionals all over the world. Currently KDE is significantly better than anything else that is open in automotive. It would just require the effort to include it and maintain it in Yocto, which is not small, and adapting Plasma a little initially, not much.

GENIVI launched a Challenge Grant Program that might help to put some funding in the equation ;-)

Whatever effort done to put Plasma on Yocto (so GDP) would also be picked up by GDP's competitor, AGL UCB (Auto Grade linux Unified Code Base), the Linux Foundation automotive group Linux (again, Yocto) based distribution. So there would be at least two players for the cost of one.

It wouldn't surprise me if Qt companies would jump in on this effort too. In order to play in the open filed today, they need to Open Source their products, which is a big risk for most of them. Playing with KDE, which is based in the technologies they are familiar with, would be simpler for them. I bet the Qt company would be heavily interested in promoting this effort. It would help to dissipate all the pushing back from the automotive industry to the current Qt license model, GPLv3 based. And it would do it in the best possible way, by providing great ready-to-use software with no competitor.

I have been one year preaching about how big the opportunity currently is for KDE, but this is not the kind of challenge that can be sustained on volunteer basis, sadly, since keeping KDE up to date in the Yocto project would require a high level of commitment from KDE as a whole. The community probably needs first a small success story and some company/corporate push before really jumping on it, I think. The support of a couple of KDE or Qt companies would catalyze the effort.

GENIVI and AGL make a significant promotion effort around the world within the automotive industry, participating in forums where KDE is unknown. Many companies that currently develop close source Qt applications for automotive would be interested in KDE which would increase our potential targets for our Corporate program.

Having KDE on GDP and AGL UCB would increase the incentive of developing new applications for many of our young developers who currently do not have automotive as a "professional target". 

Companies like LG, Renesas, Bosch, Hartman, Intel, Jaguar Land Rover, Toyota, Visteon, Fujitsu, Mitsubishi, Volvo, among many others.... are key stakeholders of GENIVI and AGL. Isn't it this attractive enough?

A success story like the one I am proposing would be yet another example of how KDE can play a key role for the Qt ecosystem. Sadly not everybody in this ecosystem understand what a great "tool" KDE can be for them. After a hit like this one, it would be undeniable.

Think about the exposure, think about where we are today... Something like this would place KDE where Unity, Android (AOSP) or GNOME are not... yet. I believe this is the kind of strategic decision that would change KDE future. But also the business perspective of those companies (specially Qt ones) who would get involved.

Let's do it now... or somebody else will.

Update(29/10/16): to find out about the state of KDE in Yocto please read the comment to this article from Samuel Stirtzel. Thanks Samuel.

Moving from a traditional product/release focused delivery model to a rolling model

The past few weeks the GDP delivery team together with some key contributors, has been working on a not very visible but still important change. The GDP project has put the basis to turn GDP release based delivery model to a "rolling" one. My colleagues will provide in a coming post the technical details behind this change. I want to provide a higher level view of what is happening and why.

Some background

GDP was born as a "demo" project. The main goal was to provide a platform to show the software components for automotive that the different GENIVI Expert Groups were developing. This was done through a delivery model focused on publishing a stable and easy to consume version of the project every few months, a major release.

Strictly speaking, GDP is a derivative. It is based on poky and uses Yocto tools to "create" the Linux based platform, adding the different components developed by the GENIVI Alliance together with upstream software. For the defined purpose, the release centric model works fine, especially if you concentrate your effort is very specific areas of the software stack with a small number of dependencies on the other areas, and a limited number of contributions and environments where the system should work.

During this 2016, the GDP has grown significantly. We have more software, more contributors, more components and more target boards to take care of. Although the above model has not been not challenged yet, it was just a matter of time.

As I explained in two previous posts [1][2], the GDP is moving from a being a Demo to a Development Platform. Changing the mission means changing the goals and the target group, which implies the need to adjust the deliverable to meet the new expectations.

So, right after the 14th AMM, the Delivery Team decided to change the delivery model to better meet the new mission, providing developers the newest possible software with the an increasing quality threshold. At the same time, in order to increase the number of contributors, the GDP needs to provide a new solid platform every once in a while. That should be done trough a solid release.

What is a rolling delivery model?

The key idea behind a modern delivery release model is to ensure that the transition from one stable release to the next one takes an affordable effort. I will put an example to picture the idea.

Problem statement

Imagine an organization that publishes one release per year. Let's assume that a particular release included 100 patches developed by employees and, during the lifetime of the release (1 year too), another 100 patches were added to the product as bug fixes and updates. At the end of the release lifetime, the product includes 200 patches that define the value the product provides to customers and users.

Either for technical or business reasons, a year later it is time to upgrade. Our organization has to create a new Linux based system with newer upstream code and they have to integrate the patches from the previous release plus the updates and bug fixes developed for the coming release.

After a simplification process done by engineers, the number of patches needed to be integrated in this newer base system is reduced to 150. The organization also wants to add to this new release another 100 patches that represent the new features they have been developing during the last year for this new version.

The delivery team now has to integrate 250 patches in the new base system, 150 of them coming from the previous release. One might think that the effort required to do this is 2.5 times the effort invested in the previous release. Maybe you think that the effort is not so high since some of the patches have been developed thinking about the new base system. There are many other considerations like this one that might affect the initial estimation. This example is obviously a simplification.

However any experienced release manager will tell you that moving patches integrated in an older system base onto a newer one (forward-porting) requires additional effort, beyond a linear relation with the number of patches. Forward-porting is the "road of hell". Iterate this example a few times and you will understand why there are so many organizations out there that have as many people focusing on delivery as they have in development. They migrated to Linux base system keeping the traditional delivery model they had while working with closed source software.

Possible solutions

One of the paths to improve the situation is upstreamming those changes that affect generic components. Some companies also upstream their new features early in their development process, generally looking for wider testing, or after they have been released to customers, to increase adoption and reduce future maintenance effort. This is definetly a must do.

From the delivery perspective, the most popular way to tackle the problem though is reducing the release cycle, so the number of patches to forward-port in each release is smaller. The development time and the maintenance cycles are also smaller. The same applies to the complexity of the forward-porting activities. "Jumping" from one release to the next one is easier to do. Add automation of repetitive tasks to this recipe and you feel you have a win.... for some time.

The journey through the "road to hell" becomes more comfortable, but our organization is still getting burned, even in the case that our customers and ourselves can digest releasing frequently. We all know how expensive and stressful a release might become.

The most suitable option to achieve sustainability while scaling up the amount of software an organization can manage without releasing more often than your market can digest is to change your delivery model.

Rolling delivery models are a serious attempt to solve this problem, putting integration as the central element instead of the software itself.

This model is not new. Gentoo has been doing it forever, but it was Arch Linux who implemented it in a way that immediately attracted the attention of thousands of developers. Still it was a model with no hope beyond hardcore Linux developers. openSUSE brought this model to a new level by implementing a process which output was stable enough for a much wider audience, and compatible with the release of a more stable and a commercial releases. Nowadays there are other interesting examples out there that commercial organizations can learn from.

What is a rolling model?

It is still hard to define but essentially it is a process in which ideally you have one continuous integration pipeline as the one an only entry point for the software you plan to ship. Releases then become snapshots of all or part of the software already integrated after going through a specific stabilization, deployment and release process.

So ideally, if you release a portfolio, you integrate only once, reducing significantly the costs of having different engineers working on different versions of the same software and forward-porting, among other benefits.

So a rolling delivery model is a lot more than a continuous integration chain, although that is the key point.

Please have in mind that this is an oversimplification. This description doesn't go into detail on other key aspects like maintenance cycles, how upstreamming affects the process, strategies towards updating the released products, etc.

A transformation process that takes an organization from a release centric model to a rolling one is about doing less and doing it faster, so less people can handle more software with less pain, allowing more people to concentrate in creating value, developing new and better software instead of just shipping it.

Back to GENIVI

Moving from a release centric to a rolling model is hard work. Frequently it is easier to start all over again. Since the GDP is still a relatively small project, we can afford going through the transformation process step by step.

The first stage has been creating that single integration chain and treating GDP-ivi9, our latest release, and those that follow it, as a deliverable of what we call today Master. Ideally, no single patch will be added directly to the release branches. They should come from Master. That way, we reduce (ideally to zero) the effort of forward-porting of patches while putting in the hands of our contributors the latest software on a regular basis.

To do so, we are in the process of adapting our simple processes and CI system to the new model, GDP repository structure, the wiki contents, the task management structures, several key policies, our communication around the project...

The GDP will face a very interesting challenge since this model needs to be proven successful for a derivative. If we are able to move fast enough, it will come the time in which we will need to decide if GDP keeps being a derivative or it becomes upstream, that is, either GDP limits the delivery speed based on the Poky release cycle, or we work upstream with the Yocto project to increase our delivery speed.

That is a good problem to have, isn't it?

If (almost) everything goes right, after adding a few needed services in GENIVI's infrastructure and ensuring the updated software is in compliance with selected verification criteria, the same number of people will be able to manage and deliver more software. And once the new processes become more stable, automation will not just increase efficiency, it will boost the project by allowing GENIVI to achieve goals that only big organizations with large delivery teams can do. This is the kind of transformation that takes time to consolidate, but has a huge impact.

Based on my experience, I believe that if GENIVI is able to sustain this effort and keep a clear direction the next couple of years, the benefits of moving towards a rolling model will be noticeable even outside the industry.

This blog post was originally published in the GENIVI blog site on 2016/07/04. I have adapted the formatting to adapt it to Blogger. The content should be the same.

Embrace Open Source culture: the 5 common transformations.

Article originally published at Linkedin on May 15th 2016.

This is a story of what I have lived or witnessed a few times so far. A story of an organization that used to consume, develop and ship proprietary software for many years. At some point in time, management took the decision of using Open Source. Like in most cases, the decision was forced by its customers, providers, competitors... and by numbers.

A painful but unavoidable transformation was required.

 

1.- Open Source consumer

Engineers had to learn a new system, adapt or re-write those features that used to made the organization unique, together with many other painful actions. It was expensive at the beginning but, due to the cost reduction in licenses and the change that Linux represented in the relation with providers, in a few years it was clearly worth it. And if it wasn't, it didn't matter since it was what the market demanded. There was no way back

Every software organization has gone through their unique journey, but the final sentence of the story has been the same for all of them: they became Open Source consumers.

 

2.- Open Source producer

This organization gained control over its production and, by consuming Open Source, it could focus many resources in differentiation, without changing the structure, development and delivery processes. At some point, it was shipping products that involved a significant percentage of generic software taken “from internet”.


It became an Open Source producer.


You can recognise such organizations because they frequently create a specific group, usually linked to R&D, in change of bringing all the innovation that is happening "in the Open Source community" into the organization.


Little by little this organisation realised that giving fast and satisfactory answers to their customer demands became more and more expensive. They got stuck in what rapidly became an old kernel or tool chain version.... Bringing innovation from “the community” required back-porting, solving complex integration issues, incompatibilities with what your provider brings, what your customer wants.


So they have to upgrade.


This organization will be able now to take advantage of all the common features and compatibility that the new kernel, the new tool chain... brings. But, guess what, forward porting all the differentiation features this organization has developed, all the bug fixes, is so much work and so complicated that the challenge put the organization at risk.

 

 

3.- Open Source contributors

The organization feels now the downsides of becoming a blind Open Source consumer and producer. Execs feels like when a bubble explodes and they are inside of it. They has less control than they thought, which turns out to be expensive, and what is worse, they lack the expertise within the organization to gain it....

But, after struggling for some time, this organization survived, which means that it has learnt some lessons:

• Upstream those features that are not differentiation factors any more.
• Increase the investment in that reduced groups of rock-stars that are up to date of what's going on "in the different communities".
• Invest in those Open Source projects that develop the key software you consume.
• Even better, start your own Open Source project to promote your technologies and be perceived as a leader...
• Reduce the upgrade cycle, so the "upgrade pain" is lower. As a side effect, the organization has the opportunity to increase the cash flow when doing two smaller upgrades instead of a big one. At the very end, the real profit comes with the first update, not with the new version, right?

This organization ended up upstreaming features when they could, normally very late, because “they do not have time”, frequently assigning that task to young inexperienced developers or, even "better", subcontracting it, which is "cheaper".

You can recognise that this organization has gone through the described process when attending to conferences given by any of its executives. They cannot stop talking about how much they contribute to this and that community, about how awesome the community is, how important it is to be open and share.... they are referring all the time to communities/upstream as "us” and “them". They think they got it, they really do.

Most of them believe they are in the crest of the wave after going through this third transformation process. They are innovative, they has been able to reduce their time to market, they are gaining reputation within a variety of communities… They are not just Open Source consumers and producers any more. They are also contributors. Some of them even heavy and "successful" contributors.

But if you look closer, they have not adopted "the Open Source way".

This organization keeps their traditional processes intact. It is managed in the same way. Decision processes are taken like when it was a proprietary company, it has not improved transparency significantly, it does not share code and practises among departments.... there is a totally different reality in front and behind its firewall, between production and R&D, between management, engineering, customer support, etc..

This organization face friction because of this reality. It still cannot move fast enough. Upgrading is still too expensive since now they have to do it more often than years ago, upstreaming goes so slow, when it happens. They cannot control the communities they are investing on...

So going through a forth transformation becomes unavoidable. Some refer to this transformation as "upstream first".

 

4.- Upstream first or becoming a good Open Source citizen

This fourth transformation basically means that upstreaming is part of your development process, not an aside task. It also means that communities are part of your delivery strategy, not an after market topic, that R&D is a two way road where you do not just consume innovation created by others but you share yours, not just "promote it". You really need to get involved.

This organization will learn that by becoming more open, their engineers learn more and faster, so the organization itself. It is at this stage where the organization really understand where the real value is in the software they produce compared to what is commodity...or that is what its executives and managers most likely believe, once again. :-)

But open source (no capital letters any more) is not about being open, but about being transparent, which means that is not just about seeing what is behind the glass, but also understand it.

I believe the fictional organization I am talking about will have to take one more step, the fifth one. It will be about "becoming upstream".

 

5.- Becoming upstream or being an open source organization

This is about understanding that, if you consume, produce and contribute Open Source, the smart thing to do is becoming an open source company. I think it is naive to pretend taking full advantage of  Open Source while keeping your traditional corporate culture, which collides with the one of those who produces most of the software you consume and ship, who are your “upstream”. You are building your business on top of them. Since you cannot control them, become "them".

The smart thing to do is to surf the wave, not fight against it, generating friction. Any manager knows that friction is expensive, reduces focus and drives away talent.  It is bad for the business.

The required culture change to succeed in this fifth transformation involves thinking less about us (company and customers) and them (community), and more about us (ecosystem). It can't be any more about upstream and downstream but about technology and service. It has to be less about upgrading and more about updating, less about "manage" and more about "lead" at every level of the organization, not just referred to execs and managers.

It is a transformation in which engineers are empowered, where management is more focused on collecting information for execs instead of producing it, and after decisions are taken, their key focus is alignment. A transformation in which execs get closer to where the real value is, to people, because they are the "masters". A transformation in which engineers not just follow, they get exposed, they take responsibility and assume the consequences... getting paid for it.

An environment in which accessing to key information does not depend on your position within the organization chart, which means that power does not depend so much  on what others ignore, but decisions are taken based on shared knowledge. A culture in which transparency is the norm not the exception.

In summary, a transformation that leads to a stage in which the organization steers its ecosystem instead of driving it. So it leads it in a sustainable way.

 

A quimera?

I understand it might sound like a quimera, but:

1. No more than it would have sounded 15 years ago any of the stories that so many CEOs or Open Source Program managers from leading corporations are telling nowadays in popular FOSS events about "their transformation".
2. I do not think the debate is if this fifth transformation will be needed, but about when and how to go through it.
3. My +10 years of experience in Open Source and +17 as manager tells me that, waiting to face any of the first four transformation processes until you have no choice is an unnecessary risk. I suspect the same will apply to the fifth one.

So my message is,

1. Consume, produce and contribute to open source being a good citizen.
2. Embrace Open Source culture... better sooner than later.

Pic link.

Testing => quality. Really?

Introduction

Nowadays the topic automated testing is becoming mainstream. Organizations and projects are investing significant effort in creating tests, using tools to automate them and plug them in their delivery chain. Combined with continuous integration tools, automate testing increases the usefulness significantly. I obviously find this trend unavoidable. Sooner or later every software organization will eventually go through it, if they have not already.

This movement is fairly new. Concepts like automate testing or continuous testing, in the context of continuous delivery, still do not have 10 years of history. We need to be careful with trends. The topic is so hot these days that the association between automated testing and quality is becoming the norm, also in Open Source.

Open Source became the winning "culture" in several industries more than five or ten years ago. Automated testing in the context of continuous delivery was not popular back then. Still, Open Source influence and adoption expanded also because of superior quality.

How come?

When I think about quality in Open Source, one key principle and three actions come to my mind.

 

Principle: transparency

Transparency is about seeing what others are doing, but also about understanding. This second part is too often forgotten.

Action 1: Code review

Transparent code review, (again, see & understand) is, in my opinion, the most powerful quality assurance measure a project or organization can apply. It is the fundamental action in what some call the FLOSS development model.

It has a side effect that I really like as manager: it improves younger developers skills. It also brings many other positive side effects.

 

Action 2: dogfooding

A few weeks ago in a workshop with a customer, Codethink CEO Paul Sherwood was explaining this point with an example that I stopped talking about several years ago. I found it so obvious that at some point I gave up fighting for it. After listening to him, not anymore. The example was.... your organization is developing Linux based products, use Linux, not Windows.

Simple, right?

Dogfooding is another of those actions that in long term Open Source projects is frequently taken for granted but that is not the norm in commercial environment. So many projects driven by newcomers to Open Source do not pay enough attention to it.

The impact over quality of dogfooding in the mid term is impossible to calculate. Still I believe is huge.

Action 3: delivery model that maximises the influence of early adopters

Who are early adopters? They are the developers or power users who like to consume experimental or pre-releases of your "product". The number of those willing to report bugs is significantly bigger in relative numbers than in consumers.

Increasing the number of early adopters, reducing the hurdles they face to use your software, analyse/debug problems and report should be a key activity among those projects worried about quality assurance. Adapting your delivery process to maximise their impact, not just have a positive effect in the use cases your software was designed for, but in others, expanding the knowledge about how your software will behave in the hands of users. Like it should happen between developers and delivery engineers, the feedback loop with early adopters should be very short, so you can provide them improved pre-releases in short cycles.

Open Source has reached the current point understanding how important the role that early adopters play is.

Personal note about this third topic

I want to make a point here before moving forward.

It seems to me that there is a new wave of Open Source projects, specially those driven by commercial organizations, that underestimate the mid term effect early adopters have on the quality of a project. I also see how the Continuous Delivery hupe, focused on the developers and delivery engineers, is leaving the early adopters behind in some cases. Specially in those Open Source projects in which the project is developed and delivered by full time dedicated engineers.

Many projects pay little attention to making their frequent releases truly installable, documented, simple to debug without complicated tools or even centralised infrastructure, bug trackers simple/fast to use, treat bug reports as a valuable asset . In summary, early adopters cannot follow the pace and, when they do, they need to spend a lot of energy to be valuable.

Let's go back to the main argument.

 

Conclusion

Code review, dogfooding and early adopters in transparent environments has been, I believe, the pillars that has made Open Source what it is today in terms of quality. And then, only then, automate testing, or continuous testing comes to place, in addition, not in substitution, not before, not in between.... in addition.

Are you doing Open Source? Don't take shortcuts. Surf the "trend wave" instead of embrace it blindly. Learn first, look carefully what sustainable projects are doing.

Quality is as much as culture as it is about having a nice dashboard full of green lights. Testing => Quality is, in general, a wrong association of ideas.

And yes, test frameworks, board farms executing thousands of tests, green lights in dashboards, etc. are awesome. Probably a forth pillar in the coming future.

Say Hi! to the new GENIVI Development Platform

On Wednesday February 17th, the GENIVI Alliance released a QEMU image of the GENIVI Demo Platform ivi9 Beta version, together with everything needed (instructions, source code, recepies, etc.) to build GDP-ivi9 with Yocto. A few weeks later, on March 8th, the first release candidate was published.

Finally, last April 19th GDP-ivi9 was published targeting QEMU, Renesas Porter and RPi2. Check the release announcement and download the different images and source code from the GDP download page.

I joined the GDP project in November 2015, leading a small team of developers from Codethink with the idea of moving GDP from a demo platform towards a collaboration platform. In summary, going from +r-- to +rwx. 

What was GDP?

GENIVI Demo Platform was the compilation of middleware components developed by GENIVI integrated with Yocto or Baserock, based on poky, designed to showcase and test the work done by GENIVI's Expert Groups.

What is GENIVI Development Platform?

At GENIVI's 14th All Members Meeting (AMM) is was announced that GDP would change his name, from Demo Platform to Development Platform, reflecting the new spirit that has arisen during the delivery of the  GDP-ivi9 version.

The general idea will be to mature those GENIVI's modules that were developed as proof of concepts (PoC) and provide up to date software together with a SDK, to attract developers to participate as contributors, having GDP as their number one Open Source platform for automotive.

Find further information about GENIVI Development Platform at GENIVI's public wiki, in the GDP project pages. The name change, recently announced will be reflected in the wiki in the coming weeks. 

Coming actions

During the coming weeks, the GDP delivery team will focus on the following topics:

• Migration from the current infrastructure to Github.
• Confluence will remain as the project wiki and JIRA as the ticketing system. The same applies for the rest of GENIVI.
• Add to our current targets another board: Intel Minnowboard
• Define together with the GDP community the roadmap for the next GDP version.
• Create a first alpha of the new version including the latest GENIVI software.

Feel free to propose enhancements or new features to GDP. The only thing you have to do is create a subtask under the ticket GDP-154, describe it and explain the benefits and potential risks/challenges. We will discuss them through the mailing list. I am looking forward of seeing Plasma 5 as part of GDP.

GENIVI 14th AMM and other events to promote GDP.

After te release of the new version, GDP maintainers and myself have been concentrated in making sure GDP was ready for  GENIVI's 14th All Members Meeting (AMM), that took place in Paris from April 25th to 29th.

I participated as speaker in 3 sessions and my colleagues at Codethink delivered a couple of Hands on Sessions about GDE-ivi9. It has been a lot of work but a good finish line for this release cycle. We will publish the slides the coming days.

A few weeks earlier I presented the GDP project at the Embedded Linux Conference (ELC), that took place in San Diego from April 4th to 6th. It was my first time at this conference and I enjoyed it. I also participated at the Collaboration Summit, invited by AGL and the Linux Foundation. I will provide some more details about these events in a later post.

I plan to attend to QtCon to promote GDP among Qt/KDE developers and to the Automotive Linux Summit, that will take place in Japan, to spread the word about this open project for automotive. I have also confirmed my presence in June 2nd at the OpenExpo, in Madrid. It will be my first event in Spain in quite some time.

Summary

It has been a very busy 6 months but very productive. Leading a small but promising Open Source project, that might have a big influence within automotive in the future, working together with my colleagues at Codethink and GDP community members, has been very interesting. I am learning a lot about this industry...by doing.