RabbitMQ Upgrade

Problem







I am currently running vFabric RabbitMQ 2.8.6 that was installed using the generic installer  vfabric-rabbitmq-server-2.8.6.tar.gz in a customized directory /apps/appuser/springsource/vfabric-rabbitmq-server-2.8.6/

Now, it is required to upgrade it to vFabric RabbitMQ 3.0.4.

** Although I am using vFabric RabbitMQ, but the same is applicable on the native RabbitMQ

Analysis

The default RPM installer will use the standard path /var/lib/rabbitmq/mnesia/ to store mnesia database that contains the queues and exchanges definitions, also the persistent messages are located under the same directory /var/lib/rabbitmq/mnesia/rabbit\@<hostname>/

The generic installer vfabric-rabbitmq-server-2.8.6.tar.gz stored mnesia database and persistent messages under a customized directory path  /apps/appuser/springsource/vfabric-rabbitmq-server-2.8.6/var/lib/rabbitmq/mnesia.

As described in the RabbitMQ 3.0.4 Release Notes, to upgrade a non-clustered RabbitMQ 2.8.6 to 3.0.4, simply install the new version 3.0.4 and all configuration and persistent message data is retained but as per our standards the new 3.0.4 installation will be at  /apps/appuser/springsource/vfabric-rabbitmq-server-3.0.4/ and it will not detect the old mnesia directory which mean none of the configuration or messages will be migrated to 3.0.4 installation.

Workaround

As a workaround, you can overcome this limitation by:

1. Check the queues and messages from the RabbitMQ 2.8.6 console http://<hostname:55672>
2. Shutdown RabbitMQ 2.8.6
3. Backup the installation directory (/apps/appuser/springsource/vfabric-rabbitmq-server-2.8.6/)
4. Install the RabbitMQ 3.0.4 to the new customized directory /apps/appuser/springsource/vfabric-rabbitmq-server-3.0.4/
5. Make sure that RabbitMQ is not running
6. Copy all files under the old mnesia directory from /apps/appuser/springsource/vfabric-rabbitmq-server-2.8.6/var/lib/rabbitmq/mnesia/rabbit@<hostname> to /apps/appuser/springsource/vfabric-rabbitmq-server-3.0.4/var/lib/rabbitmq/mnesia/rabbit@<hostname>
7. Start RabbitMQ 3.0.4
8. Check if the queues, exchanges and messages from the RabbitMQ 3.0.4 console http://<hostname:15672>
9. Your queues, exchanges, and messages will be there :)

Limitaions

This workaround is only valid on the same host server, moving mnesia database is not supported between different hosts with different hostnames.

RabbitMQ Worker Example

After RabbitMQ installation, tried simple programs to send and receive messages from a named queue.

In this example, we'll create a Work Queue that will be used to distribute time-consuming tasks among multiple workers to avoid doing a resource-intensive task immediately and having to wait for it to complete by scheduling the task to be done later.

We encapsulate a task as a message and send it to a queue. A worker process running in the background will pop the tasks and eventually execute the job. When you run many workers the tasks will be shared between them.

1. Download the Java client from download page
2. Extract the downloaded rabbitmq-java-client-bin-3.1.5.zip 
3. Copy all *.jar files from the extracted folder to C:\RabbitMQ
4. Copy NewTask.java from this link (open this link and copy the code from there to C:\RabbitMQ\NewTask.java)
5. Copy the Worker.java from this link (open this link and copy the code from there to C:\RabbitMQ\Worker.java)
6. Make sure that JAVA_HOME/bin is part of PATH
7. Run the following commands to compile NewTask.java and Worker.java

cd C:\RabbitMQ

javac -cp rabbitmq-client.jar NewTask.java Worker.java

     8. You need two consoles to run the Workers that represents the two consumers, run the following command in both consoles to start the workers:

java -cp .;commons-io-1.2.jar;commons-cli-1.1.jar;rabbitmq-client.jar Worker

     9. Run the following command to start publishing new tasks

java -cp .;commons-io-1.2.jar;commons-cli-1.1.jar;rabbitmq-client.jar NewTask

More technical details are available here.

RabbitMQ Installation

In this tutorial, we will go through the RabbitMQ installation and testing steps on local Windows 7 64-bit machine.
The steps are based on RabbitMQ 3.1.5 that was the latest release at the time of writing this tutorial.

Erlang Installation

As per described on Erlang Versions page, you will need Erlang R15B or later to run RabbitMQ on a 64bit Windows machine.

1. Download R15B02 Windows 64 Bit Binary File from Erlang R15B02 download page.
2. Double click the executable otp_win64_R15B02_with_MSVCR100_installer_fix.exe
3. Follow the wizard steps (click the screen shots to see further details).

RabbitMQ Installation

1. Download the latest available release of RabbitMQ 3.1.5 from download page
2. Double click the executable rabbitmq-server-3.1.5.exe
3. Follow the wizard steps (click the screen shots to see further details).

After the installation is completed, you will have RabbitMQ running on default port 5672
You can control RabbitMQ using the shortcuts in start menu:

Java Clients

RabbitMQ is a message broker. The principal idea is pretty simple: it accepts and forwards messages that are binary blobs of data.

In this part of the tutorial (details on how this works is available here) we'll write two programs in Java; a producer that sends a single message, and a consumer that receives messages and prints them out. It's a "Hello World" of messaging.

1. Download the Java client from download page
2. Extract the downloaded rabbitmq-java-client-bin-3.1.5.zip 
3. Copy all *.jar files from the extracted folder to C:\RabbitMQ
4. Copy the Send.java from this link (open this link and copy the code from there to C:\RabbitMQ\Send.java)
5. Copy the Recv.java from this link (open this link and copy the code from there to C:\RabbitMQ\Recv.java)
6. Make sure that JAVA_HOME/bin is part of PATH
7. Run the following command to compile Send.java and Recv.java

cd C:\RabbitMQ

javac -cp rabbitmq-client.jar Send.java Recv.java

     8. Run the following command to start the consumer:

java -cp .;commons-io-1.2.jar;commons-cli-1.1.jar;rabbitmq-client.jar Recv

    

     9. Run the following command to start the producer:

java -cp .;commons-io-1.2.jar;commons-cli-1.1.jar;rabbitmq-client.jar Send

RabbitMQ

RabbitMQ

RabbitMQ is a message broker. In essence, it accepts messages from producers, and delivers them to consumers. In-between, it can route, buffer, and persist the messages according to rules you give it.

The RabbitMQ server is an implementation of an Advanced Message Queuing Protocol (AMQP) broker. It is written on the widely-used Erlang/Open Telecom Platform (OTP), an "always available‟ platform that has been in production use in carrier grade telcos since the late 1990s. The core server is unusually compact, with only 12,000 lines of code. Because RabbitMQ implements the open AMQP protocol, it is not necessary to know Erlang, and many users prefer to manage the broker using Java or Ruby.

RabbitMQ, and messaging in general, uses these terms:

• Producing: means nothing more than sending

• Producer "P": A program that sends messages

• Queue: is the mailbox that lives inside RabbitMQ. The messages that flow through RabbitMQ and your applications can be stored only inside a queue. A queue is an infinite buffer where you can store as many messages as you like

Many producers can send messages that go to one queue - many consumers can try to receive data from one queue.

• Consuming: is receiving messages
• Consumer "C": is a program that mostly waits to receive messages

Producer, consumer, and broker do not have to reside on the same machine; indeed in most applications they don't.

Puppet

What is Puppet?

Puppet is simply an IT automation software that helps system administrators to manage infrastructure throughout its lifecycle, from provisioning and configuration to patch management and compliance and to automate repetitive tasks, quickly deploy critical applications, and proactively manage change, scaling from 10s of servers to 1000s.

The content of this post is available as Presentation with more technical details.

How Puppet Works?

Puppet uses a declarative, model-based approach to IT automation that can be summarized in 4 steps:

1. Define the desired state of the infrastructure’s configuration using Puppet’s declarative configuration language
2. Simulate configuration changes before applying
3. Enforce the deployed desired state automatically, correcting any configuration drift
4. Report on the differences between actual and desired states and any changes made enforcing the desired state

Resources

System’s configuration is a collection of resources that vary in size, complexity, and lifespan.

For example:

• File
• User account
• Software package
• Running service
• Scheduled cron job

The implementation/representation of these resources is different based on which operating system you are using. For example, you’d need a different command to start or stop a service on Windows than you would on Linux

Resource Abstraction

Similar resources can be grouped into types:

• File "readme.txt" has the same attributes (path, permissions, owner, ...) and will tend to look like all other services
• User "dave" has the same attributes (shell, group, home directory, ...) will tend to look like all other users

The description of a resource type can be separated from its implementation. You can talk about whether a service is started without needing to know what OS is in use or how to start it.

The main concept behind Puppet is to describe the desired resource states in a way that isn’t tied to a specific OS by saying "make sure this user exist and has these attributes" instead of saying “run these commands that create this user with these parameters if does not exist".

Resource Types

Puppet has many built-in resource types, and you can install even more as plugins. Each type has a different set of attributes available.

Printable cheat sheet that explains the eight most useful types is available here:

Puppet splits resources into types and providers that lets you describe resources in a way that can apply to any system:

• Types: high-level description
• Providers: platform-specific implementations

Puppet will have a definition of what state a resource should have. To sync the resource, it query the current state, compares that against the desired state, then make any necessary changes.

More to come in the next few days :)

DevOps

DevOps = Development + Operations

In a typical IT organization, many organizations divide Development and System Administration into different departments.

Development folks are driven by the changes that they are paid to accomplish based on the business needs and they create as much changes as possible. On the other side, operations folks see the changes as the enemy as the business depends on them to keep the services up and they are always motivated to resist change as it undermines stability and reliability.

Frequent changes vs Stable services vs Cost efficient

While Development departments are usually driven by user needs for frequent delivery of new features, Operations departments focus more on availability, stability of IT services and IT cost efficiency. These two contradicting goals create a "gap" between Development and Operations, which slows down IT's delivery of business value and create a disconnection between what is traditionally considered development activity and what is traditionally considered operations activity.

DevOps is software development method/capability developed by Flickr in response to the interdependence of software development and IT operations to support a business requirement of ten deployments per day and aims to help an organization to rapidly produce software products and services. Also DevOps aids in release management by standardizing development environments with easy events tracking.

Companies with release/deployment automation problems usually have existing automation but want to more flexibly manage and drive this automation. Ideally, this automation can be invoked by non-operations resources in specific non-production environments (Dev and Stage environments) where developers are given more environment control.

Goal and Drivers

The goal is to automate as much as possible different operational processes. It’s a simple calculation to make: invest in making DevOps a reality and we all should be more efficient, increasingly nimble, and less frustrated.

The adoption of DevOps is being driven by factors such as:

• Use of agile and other development processes and methodologies
• Demand for an increased rate of production releases from application and business unit stakeholders
• Wide availability of virtualized and cloud infrastructure from internal and external providers
• Increased usage of data center automation and configuration management tools

Tools and Enablers

There is always a mismatch in development and operations tooling. Take a look at the popular tools that developers use on a daily basis then take a look at the popular tools that systems administrators use on a daily basis,  it’s doubtful you’ll see much interest in using each others tools or significant integration between them.

What types of tools are required to make DevOps a reality?

• Automated infrastructure (configuration management): the process of scripting environments from installing an operating system, to installing and configuring servers on instances, to configuring how the instances and software communicate with one another. By scripting environments, you can apply the same configuration to a single node or to thousands. Puppet IT automation software is an example where Puppet uses declarative language to manage various stages of the IT infrastructure lifecycle, including the provisioning, patching, configuration, and management of operating system and application components across enterprise data centers and cloud infrastructures

• Shared version control: ensures all system artifacts are well defined, consistently shared, and up to date across the release lifecycle. Development and QA organizations draw from the same platform version, and production groups deploy the exact same version that has been certified by QA

• One step build and deployment: by scripting or automating a wide variety of tasks that software developers do in their day-to-day activities including compiling source code, packaging binary code, running tests, deployment to production systems

   

ahmed.elgenedy@startain.com

Hello World :)

Welcome to my software-related blog ... and my first post :)

• Q: Why I am blogging?
• A: Because I do believe in "Knowledge Management" and that "العلم فى الكراس ... مش فى الراس" .... So here I am trying to document all the technical topics that I am interested in hoping that they will be beneficial for others. Additionally, this will open the door for further discussions with technology experts.

• Q: Is it only software and technology related blog?
• A: Simply, yes :) ... If I changed my mind later, I will have to change the answer of this question :D

   

ahmed.elgenedy@startain.com