[HowTo] Setup N-Way Multi-Master replication with OpenLDAP

In this new post, I will explain how to setup a N-way Multi-master replication with OpenLDAP. There are many possibilities to setup a replication system with OpenLDAP. If you want to learn more about all kind of possible replication architecture, please consult this link.

The N-Way multi-master is certainly the most popular replication architecture for OpenLDAP. Multi-Master replication is a replication technique using Syncrepl to replicate data to multiple provider (“Master”) Directory servers. We don’t have the notion of “slave” or “consumer” here. All servers are the role of provider (“master”). I will use the term of provider in this document.

Advantages, valid arguments of multi-master replication

  • If any provider fails, others providers will continue to accept LDAP updates
  • It avoids  single point of failure (each server is a provider)
  • The providers can be located in several physical site
  • This architecture is very good for automatic failover
  • Provides an high availability architecture

Disadvantages, invalid arguments of multi-master replication

  • It has NOTHING to do with concept of load balancing !
  • Providers must propagate changes to all others servers. That means that the network traffic and write load spreads across all of the servers the same as for single-master
  • In the best case, server utilization and performance are identical for multi and single master replication
  • In the worst case, single master is better because of indexing can be tuned differently to optimize

 Warning: I assume that the replication is setup before any data is present. Test before with virtual machines. Backup before doing this! 

What do we do ?

As mentioned in the title of this article, we will setup an N-way multi-master replication system using OpenLDAP. In this example, I will use 3 OpenLDAP server running on Debian 7. You can try to follow this [HowTo] with your prefered distribution and more than 3 servers if you want. An important point is that we will use the 2.4 version of OpenLDAP. With this release, the LDAP server configuration is no longer in the /etc/ldap/slapd.conf file but directly in the database cn=config. We can distinguish 3 main steps:

  1. Check prerequisites
  2. Configuration of replication to replicate the configuration database (cn=config)
  3. Configuration of replication to replicate data database (hdb database)

It is really important to understand the difference between the second and third step.

ldap_schema_1

Check prerequisites

Before really starting this [HowTo], we need to have some prerequisites.

Of course, you need to have at least 3 openLDAP installed. If you don’t have, you can follow my other post : http://www.cyrill-gremaud.ch/linux/howto-install-openldap-2-4-server/

1. Install and configure NTP

NTP (Network time protocol) must be correctly configured and servers must be in sync.

Now, just configure this NTP client to contact correct NTP servers. In this example, i will use public NTP servers. My servers are located in switzerland, so I use swiss NTP server. Just modify ntp configuration file to setup correct NTP servers.

Just restart ntp client

2. The servers must be able to lookup each servers

Each servers must be able to contact the others ones (using a ICMP ping for example). To do this, you can use a local DNS or if you don’t have one (like me) you can modify the local hosts file like this:

This modification is to made on all servers of course.

Now you can try to ping each other.

3. Modifying slapd default configuration

Make sure that /etc/default/slapd have entries matching ldap1.gremaud.local, ldap2.gremaud.local, ldap3.gremaud.local or you might get this error : read_config: no serverID / URL match found. Check slapd -h arguments. This modification is to made on all server with the corresponding DNS name.

Configuring cn=config replication

All below example can be used with this command:

or with this one if you have TLS enable

I highly recommend to create a .ldif for each step of the configuration ! At the end of this document, you can download a zip file that contains all required ldif file.

To start correctly, we need to load syncprov module in each servers.

 For this first step, I give you in detail the command to use to execute it, and the output that you should see.

Always in all servers,  you need to setup the configuration database for replication. Change olcServerID for each servers. For example, for ldap1, set olcServerID to 1, for ldap2, set olcServerID to 2 and for ldap3, set to 3.

Then set a password if not already done.

It’s important to run this command on all servers and not simply copy the result value even you set the same password on all servers. Now set this password to the configuration.

At this point, you can login with ldapmodify, assuming this was done on ldap1.

You will be prompted to enter the password. If the password is incorrect, you will see a error message (ldap_bind: Invalid credentials (49)), nothing otherwise.

If you have TLS enabled and want / need to use it, you must use this command:

Note : I had some trouble to connect like this if the OS is Ubuntu server. In fact, the default configuration of slapd packet is different on Ubuntu than Debian. The olcDatabase={0}config,cn=config has no olcRootDN with Ubuntu. You must add it with this command:

Now, always in all servers, we need to add configuration replication. We need to set in the configuration which servers, using olcServerID, is concerned by this replication.

and on all servers, add the syncproc to the configuration

The last step is to add SyncRepl between the servers.

Or this one if you want to enable TLS for replication (need to have TLS enabled and configured ! Look here

Note : Sometime, an error occur when trying to add  olcMirrorMode . Look my post to stackoverflow (solution is present). My problem was that I have created my ldif file under Windows OS and ldapmodify can’t parse the character “-” (minus) on linux. If you have this problem, just create a new file with nano on linux, and edit it manually ! (no copy/past from old ldif). That should be working correctly.

Because we use the replication’s type “refreshAndPersist”, each servers will maintain a TCP connection with the 2 others servers. To check if this connection is set, you can use this command:

The lines 4 and 5 are the confirmation that the current server (ldap1) has the 2 others servers (ldap2 and ldap3) connected to it. The 2 lasts lines are the 2 output connections from ldap1 to ldap2 and 3.

Note: This test does not confirm that replication is working but only links between servers.

Now the configuration part should be working. You can test this by changing a configuration parameter on either ldap1, ldap2 or ldap3 and confirm that all servers have the same configuration. You can simply made a slapcat on each servers and check if the configuration part is the same. For example, you can execute this to ldap1 to see if the replication works.

 We just add a new fake olcServerID (4 ldap://ldapTest.test.local). If the replication works, you must see this new entry on ldap2 and ldap3. Execute the next command on ldap2 and ldap3 to check.

 If you have the line 7, your replication for cn=config works correctly. If the configuration database replicates correctly, you can do the next step.

Note: don’t forget to remove this fake entry.

Adding other databases for replication

Now it’s time to add a “data” database to replicate. It’s often that the administrators want to do: The replication of business data. Since the configuration does already replicate, we only need to do this on ONE of the servers. Just add the syncprov module to your business database HDB.

Note: It is likely that you will get errors about olcRootDN, olcRootPW and olcSuffix. If you have this, simply change “add” keyword by “replace“. You will get this error only if the parameter is already in the configuration.

Now, just add syncrepl between servers. In many distribution, when you install slapd packet, you will be prompted to configure a little the HDB database etc. In my case, I don’t have added olcSuffix, olcRootDN, olcRootPW for olcDatabase={1}hdb,cn=config because these parameters had been already set. I just need to add olcSyncRepl and olcMirrorMode.

Now just setup index for the HDB.

If everything is correct, on all of 3 providers servers, you should have new a working replication for business data (HDB).

To test if my HDB database replicates correctly, I simply make a modification in the DIT, and capture traffic with wireshark. I have made the modification on ldap2.gremaud.local (192.168.1.12).

syncExample

We can see in frames 10 and 11 that 192.168.1.12 send the new ldap entry (dc=childrens,ou=Users,dc=gremaud,dc=local) to ldap1 and ldap3 (192.168.1.11 and 192.168.1.13). Please note that each providers which receive update, will propagate this update too !

Note : It is possible that a convergence time is necessary. Wait 5 minutes and check the networks connections with this command:

You must have 8 established connection (if you have 3 servers).

  • 4 connections for cn=config (2 outbound, 2 inbound)
  • 4 connection for HDB database (2 outbound, 2 inbound)

ldap_schema_2

Errors encountered

Coming soon.

 

This [HowTo] is not very easy, so if you encounter errors, bugs or anythings else, please write me a comment because I’m not an expert with openLDAP.

LDIF sources download : download ldif files

References:

http://www.openldap.org/doc/admin24/replication.html#MirrorMode
http://www.openldap.org/doc/admin24/syncrepl.html
http://www.openldap.org/doc/admin24/replication.html#Set up the provider slapd

Share on FacebookShare on Google+Tweet about this on TwitterShare on LinkedInShare on RedditShare on StumbleUponShare on TumblrPin on PinterestFlattr the authorDigg thisBuffer this pagePrint this page

18 comments

  1. Great article but can you please do something about the background? The light grey runs out before the text does which makes it VERY hard to read over the Matrix graphic 🙁

    1. Hello Philip Colmer, excuse me for the late response. Can you send my what you see exactly ? This will help me to understand and resolve your problem. Thank you very much for your feedback. I appreciate it very much. Best Regards. Cyrill Gremaud

  2. Hi , I am trying to configure the same for multi master configuration using TLS , but the sync replication was failing between two servers , failed to start tls

    Do we need to add URI with ldaps:// instead of ldap://
    Add configuration replicationDefault
    dn: cn=config
    changetype: modify
    replace: olcServerID
    olcServerID: 1 ldap://ldap1.gremaud.local
    olcServerID: 2 ldap://ldap3.gremaud.local
    olcServerID: 3 ldap://ldap3.gremaud.local

    Can we run openldap multmaster service using TLS with uri ldaps:// only instead of ldap:// , I want to make sure the communication between two master servers should be secure, I am not able to do it without URI ldap:// in the slapd args. ? Does TLS requires port 389 to start the communication ? The reason I am asking is that the sync replication was failing If I use only ldaps in the provider URI , it is working fine If I use URI ldap:// for the provider ? Any suggestion for this scenario ?

    1. Hello shashidhar and thank you for your comment. Sorry for my late response. There is an important difference between ldaps and ldap with startTLS extension. With the first one, your server must listen on a new specific port (636 by default for ldaps) and with startLDAP extension, OpenLDAP uses the current existing ldap connection running on TCP 389 to mount a TLS session. This second alternative has the advantage to use only one port. If you sync replication fail when you using TLS, specially if you have the “failed to start tls” error in the log, that mean you have a error in your configuration. For your comment, olcServerID must use ldap:// like you use it now. With startTLS you have 2 options to establish the connection with OpenLDAP. In the parameter olcSyncRepl, you can use startTLS=critical or startTLS=yes. The difference is that with startTLS=yes, if the TLS failed to start, the communication will be done on TCP (ldap) to ensure the availability. Don’t hesitate to contact me personally using the contact form if you have more problem. Best Regards. cyrill gremaud

  3. Hi there,

    Great article, just two short notices:
    1.” Set password” section didn’t work for me.
    You wrote this:
    dn: cn=config
    changeType: modify

    dn: olcDatabase={0}config,cn=config
    add: olcRootPW
    olcRootPW: {SSHA}SGq737yNactRCyMY70TDTQs6V1wzMRD6

    This one was worked for me:
    dn: olcDatabase={0}config,cn=config
    changeType: modify
    add: olcRootPW
    olcRootPW: {SSHA}encodedpasword

    2. You advised different config password per server. That’s not works, because configuration – including the config passwords- will be mirrored.

  4. Thank you for this HowTo, you have said : ” Warning: I assume that the replication is setup before any data is present. Test before with virtual machines. Backup before doing this! ”
    What if I want to set up Multi-Master replication with OpenLDAP and I have data present in one server ?? Thank you in advance

    1. Hello @Latyyfa and thank you for your question. Firstly, I suggest you to backup all your data in order to be able to restore them in case of troubles. Then I never tried to make this replication on existing servers with data. If you can, I suggest you to make a copy of your LDAP system (it is probably a VMware) and try to do it. I will appreciate if you can write your experience here when you have finished. BR. Cyrill

      1. My case is that I have two server Master1 and Master2, I want to set replication between the two, but I want it to go in one direction , i.e : the data in Master1 goes in Master2 , and not in the other direction ( I don’t care if the data in Master2 is lost ) . data writes are in Master1 only, but the Master1 goes offline , Master2 begin to accept data writes automatically. ( I had done the failover solution but still the replication).
        It is working fine now, but I can not control the direction of replication yet.
        When I find how, I will share it here 🙂

    1. RID’s only need to be unique inside a given consumer. I.e., if the consumer has MULTIPLE syncrepl statements, then each RID must be unique. Different consumers have no concept of what RIDs other consumers use, so they do not have to be unique across consumers. But why in your case it is a problem to set unique RID ? BR. cyrill

  5. I’m running Openldap 2.4 and Ubuntu server 14.04 ,
    When I want to execute the test part ( to test if the config database is replicated )
    I got this error :
    ldap_add: Server is unwilling to perform (53)
    additional info: shadow context; no update referral
    And I’m not able to execute any other ldapmodify or ldapadd command, I always get this error.
    I’m not using TLS , and this command ‘netstat -a | egrep “:ldap” ‘ shows that the two servers can communicate ( the connection is established) .

      1. Thank you @Cyrill for your responses, this error happens when a second master is considered as “Consumer” , which means you can not make changes to it, on config database and Data database also, it can accept changes coming from the master server “Provider” only.
        And that is caused by the olcMirrorMode attribute, that should be set to TRUE.
        In my case when I set it using an ldif file :
        #
        dn: olcDatabase={0}config,cn=config # or olcDatabase={1}hdb,cn=config for Data database
        changetype: modify
        add: olcMirrorMode
        olcMirrorMode: TRUE
        #
        it does not work, (when i execute it, it does not show any error, but that didn’t add the attribute to the file). I had to change it in the specific file manually and restart slapd
        I didn’t understand why, but that solve the problem.

        1. Hi Latyyfa, In my case, I had a problem with this part too. Finally, I found that the error was here just because I edited the ldif file (or copied the content) from Windows…. Try to create a new empty document directly on linux with touch command and edit it manually with nano for example. Write all the lines by hand and do not a copy/past.

  6. Thank you very much for this guide. With some minor adjustments I made it work with our existing OpenLDAP full of data. Great work!

  7. Thank you Cyrill its a great article but ‘netstat -a | egrep “:ldap” ‘ shows ‘TIME_WAIT’ and i must restarting slapd te see this time wait, after short time i have nothing describing the connextion,any help!

  8. Many thanks for this walk-thru. I finally got repl to work thanks to your document. Getting the servers to act as mirrors sometimes needed a restart of slapd before the connections are established. I tested a reboot of one of the servers and the replication was restored witout any intervention!

Leave a Reply

Your email address will not be published. Required fields are marked *