Virtue of Necessity. Canary, sublime your company.

The past July 16th I participated in the Tenerife LAN Party, in its section Tenerife Innova, invited by the Free Software Office of La Laguna University, included in the track titled (free translation from Spanish) Open Source from the Canary Islands, stories told in First Person.

This Free Software Office is well known in Spain for managing the biggest KDE deployment in Spain with 3k computers spread in several computer labs, laboratories and libraries, among other internal projects.

My talk (20 min) had as title something like: Virtue of Necessity. Canary, sublime your company. You can find the slides (in Spanish) in my site or in my Slideshare account.

What was the talk about?

The natural growth path for a software company is creating a site, grow until it reaches a point in which, a second production site is needed. Meanwhile, departments like sales and technical support might grow distributed. As the company grows, the number of production sites grow  with it. The organization structure vary with the nature of the company, sometime production teams are replicated across different sites, sometimes different business units are divided per site and others a particular site host teams that take care of different products-(micro)services.

In general, if the company follows an "agile" approach, it will try to reduce the inter-sites communication needs by placing the team members together in a particular site. Based on my experience and how FOSS has been developed, depending on the market you are playing, turning your company into a distributed environment might be a smart move. 

Let's start providing some context and definitions.

What do I mean by sublimation in this context?

Sublimation is a state change in which a substance transit from the solid state to the gas state without going through the liquid one.

What do I mean by a distributed environment?

In most agile literature, in fact in most software development management books, distributed environments are multi-site distribution. But in Open Source, we refer to a different environment. I have came out with the following (subjective) definition:

A distributed environment/organization:

• Has no site with a significant number of employees (developers). Most of them work remotely.
• Access most (if not all) corporate application though web, not through VPN, that is, are WAN environments, not LAN.
• Uses chat-IRC (or equivalent) and video conferences as the default synchronous communication channel.
• Employees spread across several time zones.
• Multicultural environment, having English as the default language.
• Organize regular face to face sprints (as we understand it in FOSS=hackathons), maybe even a corporate event where the whole company or business unit get together.

Open Source as distributed environment

Free/Open Source Software has a geographically distributed nature. As you know, most of the relevant communities, no matter which size are we talking about, are formed by developers located in many different countries, working from home or a company site. If we take a look at the most relevant ones, they are truly global. Every tool, every process, has been designed (intentionally or not) having in mind this distributed nature.

Now that Open Source is everywhere, more and more companies are embracing it, participating on its development, collaborating in global communities. from the process perspective, they are being influenced by the Open Source way, including its distributed nature.

Many of the companies that are embracing Open Source are understanding that adapting to this new environment makes collaboration easier. It reduces the friction between community and internal processes. There is a long way to go but I believe it is unstoppable for a variety of reasons (out of the scope of the talk). We are starting to see more and more organization that are fully distributed and start-ups that are born with such structure in mind.

Canary Islands, a fragmented market.

The Canary Islands is a group of 7 islands in the Atlantic sea, with two major ones (900k people each) and 5 smaller, for a total of 2.1 million people and 11 million tourists per year. Obviously tourism is the main industry, so there are 6 international airports, two national ones and 10 harbours, half of which regularly receive big ships/cruises.

Data connectivity has improved a lot the last 10 years but, due to the difficult geography, it is unequally distributed across the islands. Even within the main islands, Tenerife and Gran Canaria, there is a significant percentage of surface with zero internet coverage.

So it is a very fragmented market and, although with first class communication infrastructure, travel across the islands takes a significant amount of time, it is expensive and connectivity might be a challenge. In general, the transportation strategy has been designed to bring people from Europe not for internal mobility.

This means that, as a software company, consolidation/growth in such market is tough, very tough, even if you focus on tourism.

Software companies there expand following the "natural" approach, which is by creating a software production centre in one of the main islands, providing support from there to the other islands. Until you consolidate your position, software companies cannot afford to have developers/technical support in the second main island. If service/support is required in one of the small islands, you simply travel there. The limitations that software companies has to face due to the market conditions, rarely allow you to create a second software development centre in the Canary Islands.

There are very few Spanish cities with daily direct planes from/to Canary Islands throughout the year. Madrid and Barcelona are the biggest markets but also the most expensive cities. The flight takes 2:30 hours to Madrid and 3 hours to Barcelona, which is a lot for European standards. So opening a second development centre in the mainland keeping the headquarters in the Canary Islands is a real challenge.

In other words, if you want to scale your business, you need to assume bigger risks than companies based in the continent, despite being a cheaper place and having plenty of professional due to the existence of two Universities.

... but,

In my talk I tried to show that all those limitations can be turned into advantages if organizations, early in their consolidation process, or even from the very beginning, adopt a distributed approach. These constrains offer a first class laboratory to experiment with some of the key variables that need to be managed when scaling up your company, while leaving aside some of the most complicated ones, related to great extend with the internationalization of the organization.

I made a call to sublimate your company, going from an "on-site" to a fully distributed state, ignoring the multi-site state. Even better, create your software company as a distributed environment since the very beginning.

Why sublimating your company in the Canary Islands?

I summarized the advantages of sublimating your company if you are based in the Canary Islands, Spain, in the following statements:

• Distributed environments adapt better to the Canary Islands environment.
• It will prepare you earlier for the internationalization phase, keeping a smaller size.
• Distributed environments adapt well to certain business models and support needs, that are becoming popular nowadays.
• If your company want or is already heavily involved in Open Source communities, adapting your internal processes to those collaborative environment you participate on is easier.
• Talent attraction is less difficult.
• Some of your fixed costs turn into production costs, that is, you gain flexibility.

Which variables will be affected by subliming your company?

These are the most relevant variables to consider:

• Cost per employee: reduction of fixed costs per employee. Increase of travel costs.
• Organization chart: from vertical to horizontal
• Human Resources policies: training, coaching, people management. From f2f to online.
• Data privacy and security: adapt to a WAN environment.
• Tools: adapted to distributed with higher latency environments
• Schedule / availability: culture shift, from presence to availability
• Development and support methodologies: from f2f conversations to remote synchronous/asynchronous channels. From agile to FOSS? Is FOSS agile?. From 8-10 hours or rotation to a higher daily production/availability window.
• Costumers relation/engagement: from tradition account management to "community" engagement management. Engineers interface customers.
• Transportation and connectivity: employees need to be connected and travelling demands will increase. Accountability/reimbursement processes will be more complex.
• Business model: your business model might need to adapt to your new distributed nature.
• Potential market: the presence of employees in new areas and the fact that your processes are adapted to distributed environments might change your target market and/or nature/size of potential customers.
• Competitors: the influence of your new distributed nature might alter your positioning against competitors.

There are more but these ones are the ones that should be considered carefully before subliming your company in the Canary Islands. As you grow, internationalization will knock at your door very soon. There are other variables to consider in that case. They are not the scope of this talk:

• Time zones
• Multiculturalism
• Language barrier
• Retributions and incentives. Cost of living.
• Fiscal and employment legislation differences
• Travel and accommodation costs and reimbursements
• Accountability and taxes
• Many more...

Summary

The Canary Islands is a tougher market than the mainland of Spain. Adopting early in the software company life cycle a distributed nature allow you to adapt better and faster to this environment, preparing you better for later stages too, specially the internationalization phase. Sublimation provides you a competitive advantage, specially if you develop Open Source and participate in open collaborative environments.

There are a number of variables that should be carefully considered though. Managing them correctly is a requirement to succeed.

openSUSE transformation step 2. The user oriented distro.

openSUSE Tumbleweed is a rolling release that was designed with a clear goal, target and metric. It was developed following a clear picture of where to go. Design it was a painful but unavoidable process that challenged an assumption established in many people mindset back in 2012. Latest/greatest and stability were incompatible. Hence, Factory, the rolling release back then, was for hardcore SUSE/openSUSE OS developers only. As rolling distros, there were more popular and better options out there.

This summary might help you to get some background.

Goals are easier to achieve if you have a good reference to beat. For those who worked in the project, Gentoo and specially Arch Linux were those references. As you can imagine, transform openSUSE required management support. We had it, specially from Roland Haidl, Operations and Communities Director at SUSE back then. He created the environment that allowed those who worked in his department to be creative.... and take risks.

Simplifying, for the new "development version", a.k.a Tumbleweed (former Factory), the goal was to implement a model that allowed us to improve the existing Factory one, based on continuous delivery. The target chosen were our core contributors (packagers fundamentally) and the metric was, in summary, to make sure that, no matter how wrong things could go after an update, you would always have a console and network, so you would be able to revert your change. In terms of the process, the resulting integration deployment processes should be transparent, not just internally but also from our community members perspective. It also needed to be simpler in order to gain contributors, not just users. And it needed to empower them to own it.

Instead of following what SUSE was doing back then, the company dedicated resources to challenge itself. As result, openSUSE Tumbleweed is today, not just the best rolling distro out there, with all what that means in terms of excitement among its contributors, but is generating higher value to SUSE, since the company have an outstanding playground at home that allows them to incorporate true innovation into their production process before their competitors do.

openSUSE is discussing nowadays to take a second step, this time focused on its user oriented version. Today is openSUSE 13.2.

In my opinion, based on the previous experience, and independently of the decision/discussion process chosen, the same steps need to be taken. They are unavoidable in any transformation process. It is necessary to define a clear goal, something short that you can explain and understand easily, a clear target and a key metric that helps to clarify the "acceptance criteria" to be used during the whole process.

Like back then, I would like to see SUSE challenging itself, putting in question well established principles within the OS industry. Again, choosing a reference would make the final picture easier to achieve.

Most openSUSE users are desktop users and sysadmin. If, as I conclude from the latest oSC15 videos and factory mailing list discussions, sysadmins are the chosen target, It would be great to see SUSE/openSUSE challenging the assumption that, through a continuous delivery process, you cannot release a stable and high quality (for the target) distribution. That stability is only achievable through a waterfall like model. I would choose CoreOS as reference. It is a project that, based on different questions, is providing innovative answers to new challenges.

I would like to see that, base on the current process (standing on the shoulders of giants) openSUSE/SUSE creates a process that "pulverize" the current mindset, deprecating many of the existing problems, focusing on solving new ones. Imagine the best of both worlds, a new paradigm of OS with the green values.

It took about a year and a half for a dedicated team to release what today is Tumbleweed. I think that this second challenge is bigger than the first one. An even bigger commitment from SUSE will be needed in order to succeed.

But if the resources are there, the creative environment is set, the right steps are followed and the openSUSE community supports the effort, there is nothing that can stop the project to achieve what today are dreams. SUSE has the talent, and the experience, to make it happen.

I wish them all the best in this new challenge.

Bitacora: impact mapping

Introduction

This is the fourth of a series of articles about Bitacora. Please read the previous ones to get full context:

1. The diary (aka bitácora): towards alignment in distributed environment.
2. Bitacora: environment definition.
3. Bitacora: personas 

Why Impact Mapping?

The steps taken so far are standard for me since long time ago. At this point though, depending on the nature of the product, I did not have a fixed procedure to follow until the creation of user stories. At some point I ended up creating use cases and relating them through mind maps.

A few months ago I read Impact Mapping, from Gojko Adzic, a reputed Agile consultant and writer. He proposes a method to go from business model to Epic/stories description that I found interesting. It fills my gap with a systematic approach using a tool I love, mind maps.

It also helps deprecating a common challenge with use cases. When you get dirty with them, it is easy to write many that are not key for the main goal of the tool. Once you have many of them, it might become tough to relate and prioritize them.

Gojko propose to get rid of use cases and go directly from impact mapping to describing the epics/user stories. I have not been able to do so. It might be that I am simply too hooked to use cases and it is only a matter of time before I follow exactly the new process. It can be also that impact mapping is rigid compared to use cases so you can only get rid of them in a second/third iteration of the impact mapping process...

In any case, for Bitacora I started with an impact mapping. Once it was mature enough, I started from scratch doing the use cases. My impact mapping ended up containing 75% percent of the use cases.... and already structured. I assume that, in a collaborative design process, this percentage will be greater. I assume also I can get better with practice. So I am confident to adopt Impact Mapping exclusively. I might loose some use cases but I gain structure. Relevant but at this point ignored use cases will probably pop up later in the design process or during development. Mistakes in the prioritization due to structure deficiencies has a bigger (negative) impact along the product life cycle.

What is an Impact Mapping?

Mr. Adzic describes in his book Impact Mapping as "a mind-map grown during a discussion facilitated by answering the following four questions: Why?, Who?, How?, What?"

1. Why: centre of the map, represent the goal of our application. It should include the desired achievemnt based on a key metric.
2. Who: first level of the mind map, it refers to actors who can influence the outcome. These are the personas. At this point of the process, it is not mandatory to describe them in detail
3. How: second level of the mind map, it refers to impact we are trying to create on the actors, that is, how the product change their behaviours.
4. What: third level of the mind map, it refers to the scope, that is, what can we do to achieve the desired impact. Depending on the dimension and complexity of your product/service, this third level will be close or far away from a user story. You can either extend the What section into sub-levels (product approach) or define epics (scrum of scrums). 

You can get further information and diagrams in the website impactmapping.org I am still unsure about the direct relation between the What section and what engineers understand as a derivable. I prefer to reserve derivable as concept to a later stage (user stories) so they are analysed in detail by the product team, together with acceptance criteria. I find this arguable though.

Bitacora Impact mapping

Once you:

• understand the root of the problem I am trying to "mitigate/solve".
• know where the idea for the solution (Bitacora) comes from. 
• get yourself in context (environment and personas).

you should be able to understand the coming Bitacora Impact Map.

If not, I have probably done something wrong, which wouldn't surprise me :-) or you are not familiar with the problem to solve. Some of the ideas like "tag page" will be described in the user stories, so don't panic, we will get there.

Since these graphs has gone through several review processes, it might happen that the partial pics are not 100% up to date. The general impact map should be though.

Center and Level 1: Why? and Who?

The center lacks the quantitative goal. Since this is an ideal exercise, I will ignore it for now. We will go down to metrics and measuring impact at the very end.

At the end of the previous article, about personas, I wrote that for a introductory analysis, we could reduce them to 5. Since I went further, I considered groups and described personas for each group since we will use them at some point.

Level 2: How?

This is the second level that defines the impact for the product team members.

 This is the second level that defines the impact for the other personas.

 As you can see, the impact is described in the following areas:

• Participation: the tool should promote an increase of interactions based on what each persona did.
• Team/project follow up: bitacora should make following up a team or project activity easier.
• Analysis: analize what the team is doing or has done should be easier with Bitacora. Self analysis is as important.
• Report: Bitacora is designed to improve reporting in bothe, vertical and horizontal direction (matrix).
• For Team Member, Bitacora should work as the central point for everyday (short term)  information.

Level 3: What?

For a easier visualization I have divided this level in three different images. The first one correspond to Product Team:

This is the third level for the Scrum Master persona:

 And this is the third image, corresponding to the other three groups:

Complete Impact Map

Here you have the complete map.

You might be wondering what are the icons for.

I mentioned earlier that one of the advantages of using Impact mappings is the structure. As a consequence, you can start to assign priorities to the different scopes which is very useful before writing down the User stories. 

Note:

The result of this process should in general be features, but in certain circumstances, at this point it could also be processes. I used the "help" icon to reflect them. his concept is arguable. I am at this point unsure if it will survive the story description.

Series of articles

1. The diary (aka bitácora): towards alignment in distributed environment.
2. Bitacora: environment definition.
3. Bitacora: personas
4. Bitacora: Impact mapping

The following article will describe the use cases I worked with so you can compare both approaches.

Bitacora: personas

Introduction

This is the third of a series of articles about Bitacora. Please read the previous ones to get full context:

1. The diary (aka bitácora): towards alignment in distributed environment.
2. Bitacora: environment definition.

The environment description is incomplete without the personas. I like this concept for many reasons but the main one is because I get extremely annoyed by the use of the word user.

When designing a product/service, it is hard to agree on what a user is, specially in engineering environments. In marketing/product environments, in general is in the DNA of those involved in product design to differentiate buyers from consumers (users). Market segmentation is understood by default. In my experience, this is not always the case in engineering environments. 

I must admit that I do not use this concept every time since again, in my experience, engineers tend to show a high resistance to this concept, specially those who have direct contact to users, that is, those who work in open and collaborative environments. They tend to consider it mainly a simplification not of a representation.

Groups

We will assume there are 4 groups/roles involved in our scenario. Each group will be represented/formed by one or several personas. To label the groups, I will use names associated with roles you probably are familiar with, which is not a good practice since it limits their potential translation to a different scenarios. I do it so it becomes easier for you to identify them. The groups, based on its interaction with Bitacora are:

1. Product team: professionals included in the team that will develop the product.
2. Customer representative: professionals in charge of the requirements/backlog. Responsible for the product.
3. Senior manager: professional in charge of the above groups.
4. Supporting actors: people that interact with the team and are needed to develop/launch the product. They can be part of the company, the open source community that develops the technology the product is based upon or the sponsor.

If this blog series would try to reflect the Bitacora design process sequentially, the number of personas, even groups, should be lower, increasing them when necessary during the design or development process.

Product team

 I have considered 4 personas:

•  Junior developer (jdev)
• Senior Developer (sdev)
• Artist (art)
• Scrum master (smast)

Artist represent non-technical profiles technical teams might include like domain experts, communication experts, etc. At this stage, defining 4 personas is not the right thing to do. Two (scrum master and the rest) would have been enough. only when going into details, the four personas will make sense.

Customer representative

I have considered two personas:

• Product owner (pdm)
• Sponsor (spon)

Like in the previous case, one persona is enough for most of the bitacora design process.

Senior manager

I have considered a single persona, the engineering Vice President, which is the most common non-executive senior manager (EVP).

 

Contributor

I have considered three personas:

• Marketeer (mark)
• FLOSS community member (cmem)
• HR representative (hrep)

The Marketeer could also be any other professional within the company that interacts with the team. The FLOSS community member could also be an external consultant/domain expert contracted to help the team.

As in the previous cases, for most initial stages of the process we will follow, the first two personas could be considered as one. The third one has a very specific and limited role in this design, based on my experience, so it could be ignored or somehow included in the senior manager persona. I will refer to it in very few (but relevant) cases.

For a close follow up you might want to download the slides.

The cliparts has been taken from clker.com and 1001freedownloads.com If

Conclusion

As you can see, I have considered 11 personas, which is a lot at this point. But I started with 5 which must be considered the main ones:

1. Engineer
2. Scrum master
3. Product owner
4. EVP
5. FLOSS community member.

So from now own I will refer to them by their names or roles. Based on your input and the process design development I might refine them.

Series of articles

1. The diary (aka bitácora): towards alignment in distributed environment.
2. Bitacora: environment definition.
3. Bitacora: personas
4. Bitacora: Impact mapping

The following one will describe the impact mapping. This section will be updated with the coming articles.

Bitacora: environment definition

Introduction

This is the second of a series of articles about Bitacora. Please read the first one to get full context.

In order to understand the perfect implementation of a Bitacora we need to go down the road of a product definition. One of the first steps is to define the environment.

Bitacora is specially designed, although not exclusively, to distributed teams. So we need to first understand what do we mean by a distributed team.

Technically speaking, we need to define the environment in which our product will live. At this point, we will take it as a hypotheses, which should be the result of studying and defining our business model. But that is out of my scope so I will use as starting point an abstraction of the different environments in which I have worked. In most of them I have worked with imperfect/limited/working versions of (the ideal) Bitacora. 

Description of the environment

We will assume the Bitacora is designed to be used in a professional environment, a company that has four offices:

• The headquarters are in Silicon Valley (UTC-8).
• The main engineering office is in New York, US (UTC-5).
• QA and services dept. are based in Hyderabad, India (UTC+5:30).
• The fastest growing office is the London, UK (UTC).

Three years ago, the company changed its growth strategy due to the difficulties they were facing to attract talent. The increasing relation with some Open Source communities acted as catalyst for this change. So now, instead of relocating people, they hire the talent wherever is available, becoming what I refer to as a distributed environment.

50% of the people hired (not including sales and customer support) the last three years are home based. They expect to increase this number up to 75% in the coming three years.

The company was originally conceived as an agile company. Every product/service has been developed following Kanban or Scrum. One of the challenges they are currently facing is how to adapt those methodologies to:

• The increasing relation with open collaborative environments.
• The distributed nature of the teams.

At the same time, the company need to face an increase of the services associated to the products they are developing. This is creating a friction between processes associated to product development and services.

Due to the increasing interaction with open source communities, it is expected that this challenge will get more complicate due to the R&D nature of the work done in these communities, that incorporates new and different processes that the company do not has control upon.

But above all, the main challenge brought by the new distributed nature detected by senior managers is the the reduction of alignment within the company which has two clear symptoms:

1. Appearance of silos.
2. Complains by senior engineers and first level managers of lack of big picture and reduced horizontal communication among teams.

A new team has been formed within the company to develop a new product, which is an evolution of a former one, based on a technology developed by an open source community. The new product is sponsored by a reputed German company that is building a complete solution for a vehicle industry leader.

This team is multidisciplinary and, following the nature of the company, heavily distributed.  It will have direct relation with the open source community the product is based on, the sponsor and several key stakeholders within the company.

Finally, since this is product considered key for the future of the company, senior management want to evaluate the work done on a regular basis.

You, reader, and I, are in charge of, using this team as test case, designing a tool that improve alignment keeping high levels of autonomy, using agile principles while reducing the impact on existing company processes to the minimum.

That tool is Bitacora.

Why this environment?

I see the above environment as the maximum common denominator of the key ones I have worked in, that is:

• Team distributed in several offices in the same time zone
• Team centralized in one office with a significant % of remote (home) work with a remote product manager, all in the same time zone.
• Team distributed in two offices and several remote team members in different time zones. 
• Team completely distributed (no office) on two correlative time zones.
• Team completely distributed across the globe in several time zones.

The possible combinations increase if we introduce culture/origin/native language/profile of the team members or sponsors, the nature of the company/product/service being developed, my role in the team and the relation of the product with collaborative environments. 

I also wanted to choose an environment that might sound familiar to my blog readers. This scenario is probably not the one where Bitacora would have the highest impact, but again, my main goal is explaining the ideal solution, not creating the best business plan.

Series of articles

1. The diary (aka bitácora): towards alignment in distributed environment.
2. Bitacora: environment definition.
3. Bitacora: personas
4. Bitacora: Impact mapping

The following one will define the personas. This section will be updated with the coming articles.

The diary (aka bitácora): towards alignment in distributed environments

Introduction

There is a virtuous circle that many think is key for any organization to achieve success:

I want to focus on autonomy and its relation with alignment.

I do not know any single smart person that do not like autonomy. Definitely I want it and most of the managers and developers I have worked with too. There is no doubt that we achieve the best results for ourselves and the organization we are involved with with great doses of autonomy.

Autonomy though, as usual, requires high levels of coordination in order to be sustainable over time. It requires mechanisms of check and balance against a plan. Take a look at Scrum, for instance. It proposes to do checks and balances against the backlog every 2 to 6 weeks aprox., at the beginning/end of every sprint And it forces also to stablish a daily check and balance against the micro plan every day. That is a lot but it is the price to pay for getting autonomy.

And this is because the success of a group completely depends on the alignment of its members. Our tendency of "taking our own way" (entropy) is so high that organizations need people and many processes just to make sure everybody fight hard against it.

Definition and a little bit of history

I think it was 2004 when I started my relation with Fotón S.I. crew. Fotón is an open source company from Gran Canaria, founded in 1998, lead by Mike Vazquez and, back then, also by Gonzalo Aller.

Working with Fotón SI had a huge impact on the way I perceive management. It worked as a catalyst. I learned with them in a few months what takes ages for others.

Fotón was formed by a group of very talented and young hackers. The decision processes were very participative and transparent. Let me put you one single example: the basic accounting of the company was open to every employee, including salaries.

In their peak times they were around 20 people but it was strange to see more than 5/6 in the office. Most fotonians worked from home most of the time. They were based in Gran Canaria and my little company was in Tenerife. It was a distributed environment.

Going back to Foton, a key variable that balanced autonomy and alignment  in that distributed environment was THE DIARY, a simple concept but very effective.

Most people right now are probably thinking about that little notebook that needed a key to be opened where some boys or girls write their thoughts at early ages. Obviously that idea has little to do with what I am talking about. The term bitácora, a short way in Spanish to refer to the ship's log, is closer to what I am talking about.

A bitácora is a log chronologically organized, written by the ship's captain, kept in a wel define place where the most relevant events related with the ship, the crew and the journey where described. It had two main goals:

• Analysis.
• Report.

By writing on regular basis what happened in the ship, captains were creating a tool that allowed them to learn from past experiences. It was a very precious treasure, so preious that needed to be destroyed in case the ship got into the wrong hands. It was way more than the Black box of a plane. It captures the most important knowledge. If the captain got sick, for instance, or died in combat, his replacement used the bitácora to analyse the past history. Many of you are familiar with this concept since it is popular among scientists, archaeologists, etc.

The bitácora worked also as a reporting tool. After a few months sailing, it was impossible for any captain or officer to remember details about the trip if they were not written in real time. Providing a report about the trip was impossible without the bitácora. One interesting point was that, once written, no entry could be modified. If the captain made a mistake, he needed to create a new entry correcting the mistake, like in a check-book.

Fotón was a pret-a-porter development company organized per project. Each project had its own diary/bitácora. Following its culture, everybody had +xrw permissions in every bitácora. Agreed labels were used for different purposes: identify each user, assign relevance a specific entry through colours, etc. This idea was implemented in a hacked Twiki, integrated with Request Tracker and a mail service for notifications. A very smart, geeky, simple and efficient implementation for the time. Like all the wikies back then, UX was not the main feature but for console lovers.....that was no issue.

So Fotón extended the concept of the bitácora to a new level, going from an individual oriented tool to a group one, that is, they took a collaborative approach with a great result.

I still remember the day they explained and showed it to me. I was amazed, excited and cautious about its scalability. I was already used to write regularly what I was doing so the main hurdle did not applied to me: I had the habit. The collaborative approach was new to me though, together with the implementation.

How come something so simple has such a high impact? What makes reading what others do, think or feel so valuable? What is the relation of writing what you do and being efficient as a team? What a diary has to do with alignment? And here is the question that most people that face this tool/process for the first time ask: isn't it the diary a tool to control me?

Let me start with the last one. The bitácora is a tool to control yourself. If you think you do not need to regularly check and balance your actions against your plans, your colleagues expectations and your company goals is simply because you are not senior enough, period. The diary is just a way to achieve that, like the stand up meetings or the burn down chart.

You decide what kind of information you should include in the bitácora. Here is the main rule. Knowing that is open to those you work with, add what you think is :

• Relevant to you.
• Relevant to your peers.
• Relevant to those you interact with like, managers, other teams, customers, etc.

Do not add information that only one person should know about or that you would not say in a team meeting. Simply use your common sense. For instance, if I am frustrated about something.... I add it in the diary if I want to share it.

Fotón used to open the diary about a specific project to customers and third parties involved on it and it worked beautifully. Of course once in a while somebody wrote something inappropriate, but the benefits of these diaries were so high that assuming the damage was out of question. It became an outstanding engagement tool, as good as the best sprint planning meeting.

Very soon I experienced myself the benefits of getting the habit of writing regularly in those project diaries I was involved with, reading what others wrote and interacting with them through the diary. I got a sense of control of my day to day work and, even better, of what everybody else was doing. Control in a good way. Sync meetings were reduced, asking for weekly reports made no sense,... we were a more efficient, coordinated... better aligned organization, even in our distributed environment, despite working in several projects at the same time, with different customers, third party companies or different working schedule.

Conclusion

In many ways Fotón diaries laid upon two ideas that today are very popular: semantic and social (in a twitter way). I believe that the diary is a key complement to agile methodologies, specially in distributed environments.

In my next article I will describe in detail the concept and what the perfect implementation should look like, based on my experience since I have used it widely since then.

Series of articles

1. The diary (aka bitácora): towards alignment in distributed environment.
2. Bitacora: environment definition.
3. Bitacora: personas
4. Bitacora: Impact mapping

The following one will define the personas. This section will be updated with the coming articles.

Apply agile methodologies to upstream development environments.... if you can.

 Introduction

When the Agile Manifesto became popular and based on them, agile methodologies like Scrum, XP or Kanban, upstream development was in its early stages as collaboration ecosystems of companies.

Only a few for profit organizations embraced developing upstream back then. Most of them were small and heavily influenced by FLOSS engineers vision. Free software communities were basically driven on personal basis or the very lucky ones, together with "sponsored developers". In general, these ecosystem were not part of companies strategies.

Today, more and more companies are getting fully involved in community projects as stakeholders, not just consumers or simple contributors.

They frequently start as consumers, then, little by little they become "upstreamers", that is, they share/publish their code with the goal to have it merged (upstream code). Not without effort, many of them become successful contributors.  After some time, some of them end up understanding that is "cheaper" to play by the project rules. In summary, they learn to become good citizens.

A subgroup of the above companies end up including these collaboration ecosystems as part of their own strategies, going from contributors to  key stakeholders. A necessary step to achieve this goal is to work upstream.

Walking this path present many challenges. One of the toughest ones is related with the differences in development methodologies used internally (mostly agile) and those used in the collaboration ecosystems.

There are two fundamental variables that, in my opinion, determine this challenge:

1. Environment
2. Culture

Challenges

1.- Environment

There are two dependent variables that were not taken into account (or just partially) when the agile methodologies were defined, that are relevant in upstream development:

1. Community projects are global environments, that is, contributors are located in different "offices", frequently in different time zones.
2. Probably due to the original amateur condition of early contributors, together with the "distributed condition", the development processes (so the tools) in most mature community projects, consider, manage and tolerate high levels  of latency.  "Real time" is restricted to IRC discussions and events/conferences.

These two factors has made open source what it is today. They have been "success factors".

Agile methodologies do not embrace "distribution" environments. The widely accepted recommendation is that teams should share a physical space. It is way more than a recommendation. It is somehow a requirement.

The second case, "latency", is considered by agile methodologies as a waste. It is not tolerated.

2.- Culture

Free Software was born as a reaction to a system that promoted corporation interests over developers, so users. The agile movement was a reaction to those methodologies that put process first, not people. Hence, it is obvious that both movements share a lot: people first. 

This is reflected by some when saying that FLOSS development is agile.

In my opinion, there is a big difference between what agile methodologies and what Open Source development propose in terms of principles.

Agile methodologies promotes a strong team culture. Open Source was born "based on champions". FLOSS culture normally applies the meritocracy concept to individuals.  Open Source projects are organized around contributors, around specialists, not around teams, as we understand them in corporate environments.

This is no surprise since Agile was born in companies/corporations and Open Source was born as a viral movement, grown "by aggregation".

The conflict

In my opinion, the more the industry embrace open source, and as result, open collaboration, the higher the conflict developers and managers will face due to the above challenges.  Companies are becoming more distributed environments and are working more and more upstream, instead of simply being consumers or occasional contributors.

In consequence, it would not surprise me if we hear more and more about  "corporate development methodologies" (a.k.a. agile) vs. "upstream development methodologies" (a.k.a. FLOSS).

Scrum, XP, Kanban -ish fans will need to face those challenges and find solutions in order to succeed in open collaboration environments. In the same way, based on the increasing influence that companies are gaining in these ecosystems, FLOSS methodologies in a few years will differ from what we knew 10 years ago.

This conflict will not be (is) about a R&D vs a product/service vision, it is not about creativity vs efficiency, it is not about micromanagement vs autonomy or teams of juniors vs specialists either. It is about methodologies applied to specific environments and its limitations. Maybe a simple update of the most successful agile methodologies will do the job.... or maybe we need to revisit some of the principles.

If you got here, maybe you want to take an extra step and answer these questions. I would appreciate it:

1. Do you perceive this conflict as I do?
2. Am I missing other key elements in the diagnosis?
3. How do you think we can adapt agile methodologies so they can be adapted to FLOSS environments?
4. I am interested in knowing how you adapt agile methodologies to overcome the above challenges. I plan to write about my experience these coming days.

Where the corporate and the upstream worlds meet.... or collide.

From the corporate world I frequently hear how hard it is to predict and track what upstream developers do. On the other side, developers that work part or full time  upstream frequently underestimate the need for communicating what they do in a way that enable others (or themselves) to provide deadlines and effort estimations. Upstream and product "time lines" and cultures often differ too much to be compatible under the same environment.

Developers that come from the "product" side of the story often refer to agile methodologies like Scrum as a good approach to solve this collision. It is unclear though the applicability of this and similar methodologies to highly distributed environments, where latency is high. Although most accept today that FLOSS development is agile, I find hard to assume that we are close to find good answers when it comes to development methodologies and upstream.

There are several different scenarios I have worked on where I have to question everything I knew up until then about this particular topic. In a couple of occasions, with a bigger or smaller community, the people I directly worked with were upstream. This allowed us to define up to a great extend the methodology we worked with.

In Linaro now this is the case for a particular project. But in general, in my Group we work upstream (in the Linux Kernel), that is, we cannot define the methodology since it comes defined by the project itself. It is not the first time I face this situation although I never did at this scale.

When you are upstream, the methodology used by the core team, together with how the project work packages and workloads are organised heavily influences the success in getting contributions from third parties. Together with other key variables, how you manage latency determines how many people can follow you and potentially contribute. In summary, the faster you move the harder it becomes for contributors to collaborate with you.

Hence agile methodologies come with a high risk: isolation. And this is obvious by just analyzing the prerequisites associated with the methodology itself.

Obviously this does not mean that scrum and other methodologies are not great ones. I am just pointing challenges associated with apply them when working in a community that you drive. You need to balance the efficiency of your team with the contributions you might potentially loose from externals.

The challenge is even bigger when you are just a participant in a community. In some way, your team goes beyond your colleagues at work. Latency is frequently too high to even consider yourself and the people that works close to you "a team" in the classical sense of the word.

If your own team is distributed, even if they are in the same time zone, then we are talking about the real deal, the master challenge for managers and developers. 

Imagine now that your team is formed by 10 people distributed in 6 different time zones in 3 different continents. Then you take one of those cool books that tries to explain how to create great software using this or that methodology and, in order for you to finish it, the writer must be a really good one or you really need to be an avid reader :-) 

Still you need to make plans, to predict when a feature will be finished, when it will be merged, you still need to manage dependencies, expectations, budgets.... you still are tight to somebody else needs and requirements. You still need to fulfil expectations despite the fact your control over the environment is limited. You have team mates that depend on your work, that needs to know what you are doing and how.....  you are still part of a business no matter if you develop upstream or not.

How can you make compatible the upstream development processes used in the community you are contributing to with the way your company works? And how about your customers and partners, if you have direct relation with them? How can you take the good things that agile development propose and apply them to an environment with high latency, where one of your bigger challenges is to have an efficient team meeting, since people are distributed across the planet?

At Linaro, the engineering management team, together with our PMO, are taking a close look at this challenge, in order to iterate from our current system to a more efficient one, better adapted to our new reality, after our significant growth during the past 18 months. The idea is to find a place where corporate and community processes meet... without colliding. You cannot stop adapting, innovating, trying new things..... or you pay a high price.

One of the great things about Linaro is that we are a unique environment so we can innovate at various levels, not just at the technology one.

Open Sorce Forum at CebIT

For those of you attending to CebIT, let me tell you that I am giving a talk about what we do at Linaro on Monday March 16th at the Open Source Forum. It is the first time I talk about my current job in public so I am very excited about it. It will be also my first time in CebIT so double doses of excitement. And yes, I will talk a little bit about Linaro new initiative, 96boards.

Core Dump

The past months we have published a few articles about what we do at Core Development Group at Linaro. Our blog is called Core Dump. Check it out.