MySQL Cluster is a high-availability, high-redundancy version of MySQL
adapted for the distributed computing environment. It uses the
NDB Cluster storage engine to enable running
several MySQL servers in a cluster. This storage engine is available and in
binary releases from MySQL-Max 4.1.3. Beginning with MySQL 4.1.10a, it is
also available in RPMs compatible with most modern Linux distributions. (If
you install using RPM files, note that both the
mysql-server and mysql-max
RPMs must be installed to have MySQL Cluster capability.)
The operating systems on which MySQL Cluster is currently available are
Linux, Mac OS X, and Solaris. (Some users have reported success with running
MySQL Cluster on FreeBSD and HP-UX, although these platforms are not yet
officially supported by MySQL AB.) We are working to make Cluster run on all
operating systems supported by MySQL, including Windows, and will update
this page as new platforms are supported.
This chapter represents a work in progress, and its contents are subject to
revision as MySQL Cluster continues to evolve. Additional information
regarding MySQL Cluster can be found on the MySQL AB Web site at
http://www.mysql.com/products/cluster/.
MySQL Cluster is a technology that enables
clustering of in-memory databases in a shared-nothing system. The
shared-nothing architecture allows the system to work with very
inexpensive hardware, and without any specific requirements on hardware
or software. It also does not have any single point of failure because
each component has its own memory and disk.
MySQL Cluster integrates the standard MySQL server with an in-memory
clustered storage engine called NDB. In our
documentation, the term NDB refers to the
part of the setup that is specific to the storage engine, whereas “MySQL
Cluster” refers to the combination of MySQL and the
NDB storage engine.
A MySQL Cluster consists of a set of computers, each running a number of
processes including MySQL servers, data nodes for NDB Cluster,
management servers, and (possibly) specialized data access programs. The
relationship of these components in a cluster is shown here:
All these programs work together to form a MySQL Cluster. When data is
stored in the NDB Cluster storage engine,
the tables are stored in the data nodes. Such tables are directly
accessible from all other MySQL servers in the cluster. Thus, in a
payroll application storing data in a cluster, if one application
updates the salary of an employee, all other MySQL servers that query
this data can see this change immediately.
The data stored in the data nodes for MySQL Cluster can be mirrored; the
cluster can handle failures of individual data nodes with no other
impact than that a small number of transactions are aborted due to
losing the transaction state. Because transactional applications are
expected to handle transaction failure, this should not be a source of
problems.
By bringing MySQL Cluster to the Open Source world, MySQL AB makes
clustered data management with high availability, high performance, and
scalability available to all who need it.
NDB is an in-memory storage
engine offering high-availability and data-persistence features.
The NDB storage engine can be configured with a range of failover and
load-balancing options, but it is easiest to start with the storage
engine at the cluster level. MySQL Cluster's NDB storage engine contains
a complete set of data, dependent only on other data within the cluster
itself.
The cluster portion of MySQL Cluster is currently configured independently
of the MySQL servers. In a MySQL Cluster, each part of the cluster is
considered to be a node.
Note: In many contexts, the
term “node” is used to indicate a computer,
but when discussing MySQL Cluster it means a
process. There can be any number of nodes on a single
computer, for which we use the term cluster
host.
There are three types of cluster nodes, and in a minimal MySQL Cluster
configuration, there will be at least three nodes, one of each of these
types:
The management node (MGM
node): The role of this type of node is to manage the other nodes
within the MySQL Cluster, such as providing configuration data,
starting and stopping nodes, running backup, and so forth. Because
this node type manages the configuration of the other nodes, a node
of this type should be started first, before any other node. An MGM
node is started with the command
ndb_mgmd.
The data node: This is the
type of node that stores the cluster's data. There are as many data
nodes as there are replicas, times the number of fragments. For
example, with two replicas, each having two fragments, you will need
four data nodes. It is not necessary to have more than one replica.
A data node is started with the command
ndbd.
The SQL node: This is the
node that accesses the cluster data. In the case of MySQL Cluster, a
client node is a traditional MySQL server that uses the
NDB Cluster storage engine. An SQL node is typically started
with the command mysqld --ndbcluster
or by using mysqld
with the ndbcluster option added to
my.cnf.
Important: It is not realistic
to expect to employ a three-node setup in a production environment. Such
a configuration provides no redundancy; in order to benefit from MySQL
Cluster's high-availability features, you must use multiple data and SQL
nodes.
Configuration of a cluster involves configuring each individual node in
the cluster and setting up individual communication links between nodes.
MySQL Cluster is currently designed with the intention that data nodes
are homogeneous in terms of processor power, memory space, and
bandwidth. In addition, to provide a single point of configuration, all
configuration data for the cluster as a whole is located in one
configuration file.
The management server (MGM node) manages the cluster configuration file
and the cluster log. Each node in the cluster retrieves the
configuration data from the management server, and so requires a way to
determine where the management server resides. When interesting events
occur in the data nodes, the nodes transfer information about these
events to the management server, which then writes the information to
the cluster log.
In addition, there can be any number of cluster client processes or
applications. These are of two types:
Standard MySQL clients:
These are no different for MySQL Cluster than they are for standard
(non-Cluster) MySQL. In other words, MySQL Cluster can be accessed
from existing MySQL applications written in PHP, Perl, C, C++, Java,
Python, Ruby, and so on.
Management clients: These
clients connect to the management server and provide commands for
starting and stopping nodes gracefully, starting and stopping
message tracing (debug versions only), showing node versions and
status, starting and stopping backups, and so on.
15.2.1. MySQL Cluster Nodes, Node Groups, Replicas, and Partitions
This section discusses the manner in which MySQL Cluster divides and
duplicates data for storage.
Central to an understanding of this topic are the following concepts,
listed here with brief definitions:
(Data) Node: An
ndbd process, which
stores a
replica —that is, a copy of the
partition (see below) assigned to the
node group of which the node is a member.
Each data node should be located on a separate computer. While it is
also possible to host multiple
ndbd processes on a
single computer, such a configuration is not supported.
It is common for the terms “node” and “data
node” to be used interchangeably when referring to an
ndbd process; where mentioned, management (MGM)
nodes (ndb_mgmd
processes) and SQL nodes (mysqld
processes) are specified as such in this discussion.
Node Group: A node group
consists of one or more nodes, and stores partitions, or sets of
replicas (see next item).
Note: Currently, all node
groups in a cluster must have the same number of nodes.
Partition: This is a
portion of the data stored by the cluster. There are as many cluster
partitions as nodes participating in the cluster. Each node is
responsible for keeping at least one copy of any partitions assigned
to it (that is, at least one replica) available to the cluster.
A replica belongs entirely to a single node; a node can (and usually
does) store several replicas.
Replica: This is a copy
of a cluster partition. Each node in a node group stores a replica.
Also sometimes known as a partition replica.
The number of replicas is equal to the number of nodes per node
group.
The following diagram illustrates a MySQL Cluster with four data nodes,
arranged in two node groups of two nodes each; nodes 1 and 2 belong to
node group 0, and nodes 3 and 4 belong to node group 1. Note that only
data (ndbd) nodes are
shown here; although a working cluster requires an
ndb_mgm process for
cluster management and at least one SQL node to access the data stored
by the cluster, these have been omitted in the figure for clarity.
The data stored by the cluster is divided into four partitions, numbered
0, 1, 2, and 3. Each partition is stored — in multiple copies — on the
same node group. Partitions are stored on alternate node groups:
Partition 2 is stored on .
Partition 0 is stored on node group 0; a
primary replica (primary copy) is stored on node 1, and a
backup replica (backup copy of the partition) is stored on node
2.
Partition 1 is stored on the other node group (node group 1); this
partition's primary replica is on node 3, and its backup replica is
on node 4.
Partition 2 is stored on node group 0. However, the placing of its
two replicas is reversed from that of Partition 0; for Partition 2,
the primary replica is stored on node 2, and the backup on node 1.
Partition 3 is stored on node group 1, and the placement of its two
replicas are reversed from those of partition 1. That is, its
primary replica is located on node 4, with the backup on node 3.
What this means regarding the continued operation of a MySQL Cluster is
this: so long as each node group participating in the cluster has at
least one node operating, the cluster has a complete copy of all data
and remains viable. This is illustrated in the next diagram.
In this example, where the cluster consists of two node groups of two
nodes each, any combination of at least one node in node group 0 and at
least one node in node group 1 is sufficient to keep the cluster “alive”
(indicated by arrows in the diagram). However, if
both nodes from
either node group fail, the
remaining two nodes are not sufficient (shown by the arrows marked out
with an
X); in either case, the
cluster has lost an entire partition and so can no longer provide access
to a complete set of all cluster data.
This section is a “How-To” that describes the
basics for how to plan, install, configure, and run a MySQL Cluster.
Whereas the examples in
Section 15.4, “MySQL Cluster Configuration” provide more in-depth
information on a variety of clustering options and configuration, the
result of following the guidelines and procedures outlined here should
be a usable MySQL Cluster which meets the
minimum requirements for availability and safeguarding of
data.
This section covers hardware and software requirements; networking issues;
installation of MySQL Cluster; configuration issues; starting, stopping,
and restarting the cluster; loading of a sample database; and performing
queries.
Basic Assumptions
This How-To makes the following assumptions:
The cluster setup has four nodes, each on a separate host, and each
with a fixed network address on a typical Ethernet as shown here:
Node
IP
Address
Management (MGM) node
192.168.0.10
MySQL server (SQL) node
192.168.0.20
Data (NDBD) node "A"
192.168.0.30
Data (NDBD) node "B"
192.168.0.40
This may be made clearer in the following diagram:
Note: In the interest of
simplicity (and reliability), this How-To uses only numeric IP
addresses. However, if DNS resolution is available on your network,
it is possible to use hostnames in lieu of IP addresses in
configuring Cluster. Alternatively, you can use the
/etc/hosts file or your operating system's equivalent for
providing a means to do host lookup if such is available.
Each host in our scenario is an Intel-based desktop PC running a
common, generic Linux distribution installed to disk in a standard
configuration, and running no unnecessary services. The core OS with
standard TCP/IP networking capabilities should be sufficient. Also
for the sake of simplicity, we also assume that the filesystems on
all hosts are set up identically. In the event that they are not,
you will need to adapt these instructions accordingly.
Standard 100 Mbps or 1 gigabit Ethernet cards are installed on each
machine, along with the proper drivers for the cards, and that all
four hosts are connected via a standard-issue Ethernet networking
appliance such as a switch. (All machines should use network cards
with the same throughout. That is, all four machines in the cluster
should have 100 Mbps cards
or all four machines should
have 1 Gbps cards.) MySQL Cluster will work in a 100 Mbps network;
however, gigabit Ethernet will provide better performance.
Note that MySQL Cluster is not
intended for use in a network for which throughput is less than 100
Mbps. For this reason (among others), attempting to run a MySQL
Cluster over a public network such as the Internet is not likely to
be successful, and is not recommended.
For our sample data, we will use the world
database which is available for download from the MySQL AB Web site.
As this database takes up a relatively small amount of space, we
assume that each machine has 256MB RAM, which should be sufficient
for running the operating system, host NDB process, and (for the
data nodes) for storing the database.
Although we refer to a Linux operating system in this How-To, the
instructions and procedures that we provide here should be easily
adaptable to either Solaris or Mac OS X. We also assume that you already
know how to perform a minimal installation and configuration of the
operating system with networking capability, or that you are able to
obtain assistance in this elsewhere if needed.
One of the strengths of MySQL Cluster is that it can be run on commodity
hardware and has no unusual requirements in this regard, other than for
large amounts of RAM, due to the fact that all live data storage is done
in memory. (Note that this is subject to change, and that we intend to
implement disk-based storage in a future MySQL Cluster release.)
Naturally, multiple and faster CPUs will enhance performance. Memory
requirements for Cluster processes are relatively small.
The software requirements for Cluster are also modest. Host operating
systems do not require any unusual modules, services, applications, or
configuration to support MySQL Cluster. For Mac OS X or Solaris, the
standard installation is sufficient. For Linux, a standard, “out
of the box” installation should be all that is necessary. The
MySQL software requirements are simple: all that is needed is a
production release of MySQL-max 4.1.3 or newer; you must use the
-max version of MySQL to have Cluster
support. (See
Section 5.3, “The mysqld-max
Extended MySQL Server”.) It is not necessary to compile MySQL
yourself merely to be able to use Cluster. In this How-To, we assume
that you are using the
-max binary appropriate to your Linux,
Solaris, or Mac OS X operating system, available via the MySQL software
downloads page at
http://dev.mysql.com/downloads/.
For inter-node communication, Cluster supports TCP/IP networking in any
standard topology, and the minimum expected for each host is a standard
100 Mbps Ethernet card, plus a switch, hub, or router to provide network
connectivity for the cluster as a whole. We strongly recommend that a
MySQL Cluster be run on its own subnet which is not shared with
non-Cluster machines for the following reasons:
Security: Communications
between Cluster nodes are not encrypted or shielded in any way. The
only means of protecting transmissions within a MySQL Cluster is to
run your Cluster on a protected network. If you intend to use MySQL
Cluster for Web applications, the cluster should definitely reside
behind your firewall and not in your network's De-Militarized Zone (DMZ)
or elsewhere.
Efficiency: Setting up a
MySQL Cluster on a private or protected network allows the cluster
to make exclusive use of bandwidth between cluster hosts. Using a
separate switch for your MySQL Cluster not only helps protect
against unauthorized access to Cluster data, it also ensures that
Cluster nodes are shielded from interference caused by transmissions
between other computers on the network. For enhanced reliability,
you can use dual switches and dual cards to remove the network as a
single point of failure; many device drivers support failover for
such communication links.
Each MySQL Cluster host computer running data or SQL nodes must have
installed on it a MySQL-max binary. For management nodes, it is not
necessary to install the MySQL server binary, but you do have to install
the MGM server daemon and client binaries (ndb_mgmd
and ndb_mgm,
respectively). This section covers the steps necessary to install the
correct binaries for each type of Cluster node.
MySQL AB provides precompiled binaries that support Cluster, and there
is generally no need to compile these yourself. Therefore, the first
step in the installation process for each cluster host is to download
the file
mysql-max-4.1.21-pc-linux-gnu-i686.tar.gz
from the MySQL
downloads area. We assume that you have placed it in each machine's
/var/tmp directory. (If you do require a custom binary, see
Section 2.9.3, “Installing from the Development Source Tree”.)
RPMs are also available for both 32-bit and 64-bit Linux platforms; as
of MySQL 4.1.10a, the -max
binaries installed by the RPMs support the
NDBCluster storage engine. If you choose to
use these rather than the binary files, be aware that you must install
both the -server
and -max packages on all machines that are
to host cluster nodes. (See
Section 2.4, “Installing MySQL on Linux”, for more information about
installing MySQL using the RPMs.) After installing from RPM, you will
still need to configure the cluster as discussed in
Section 15.3.3, “Multi-Computer Configuration”.
Note: After completing the
installation, do not yet start any of the binaries. We will show you how
to do so following the configuration of all nodes.
Storage and SQL Node Installation
On each of the three machines designated to host storage or SQL nodes,
perform the following steps as the system
root user:
Check your /etc/passwd and
/etc/group files (or use whatever
tools are provided by your operating system for manging users and
groups) to see whether there is already a
mysql group and
mysql
user on the system. Some OS distributions create these as part of
the operating system installation process. If they are not already
present, create a new
mysql user group, and then add a
mysql user to this group:
shell> groupadd mysql
shell> useradd -g mysql mysql
The syntax for useradd
and
groupadd may differ
slightly on different versions of Unix, or they may have different
names such as
adduser and
addgroup.
Change location to the directory containing the downloaded file,
unpack the archive, and create a symlink to the
mysql-max directory named
mysql. Note that the actual file and
directory names will vary according to the MySQL version number.
shell> cd /var/tmp
shell> tar -xzvf -C /usr/local mysql-max-4.1.21-pc-linux-gnu-i686.tar.gz
shell> ln -s /usr/local/mysql-max-4.1.21-pc-linux-gnu-i686 /usr/local/mysql
Change location to the mysql directory
and run the supplied script for creating the system databases:
shell> cd mysql
shell> scripts/mysql_install_db --user=mysql
Set the necessary permissions for the MySQL server and data
directories:
shell> chown -R root .
shell> chown -R mysql data
shell> chgrp -R mysql .
Note that the data directory on each machine hosting a data node is
/usr/local/mysql/data. We will use this piece of information
when we configure the management node. (See
Section 15.3.3, “Multi-Computer Configuration”.)
Copy the MySQL startup script to the appropriate directory, make it
executable, and set it to start when the operating system is booted
up:
(The startup scripts directory may vary depending on your operating
system and version — for example, in some Linux distributions, it is
/etc/init.d.)
Here we use Red Hat's chkconfig
for creating links to the startup scripts; use whatever means is
appropriate for this purpose on your operating system and
distribution, such as update-rc.d
on Debian.
Remember that the preceding steps must be performed separately for each
machine on which a storage or SQL node is to reside.
Management Node Installation
Installation for the management (MGM) node does not require installation
of the mysqld binary. Only
the binaries for the MGM server and client are required, which can be
found in the downloaded archive. Again, we assume that you have placed
this file in /var/tmp.
As system root (that is, after using
sudo,
su root, or your system's equivalent for temporarily
assuming the system administrator account's privileges), perform the
following steps to install ndb_mgmd
and
ndb_mgm on the Cluster
management node host:
Change location to the /var/tmp
directory, and extract the ndb_mgm
and
ndb_mgmd from the
archive into a suitable directory such as
/usr/local/bin:
shell> cd /var/tmp
shell> tar -zxvf mysql-4.1.21-pc-linux-gnu-i686.tar.gz
shell> cd mysql-4.1.21-pc-linux-gnu-i686
shell> cp /bin/ndb_mgm* /usr/local/bin
(You can safely delete the directory created by unpacking the
downloaded archive, and the files it contains, from
/var/tmp once
ndb_mgm and
ndb_mgmd
have been copied to the executables directory.)
Change location to the directory into which you copied the files,
and then make both of them executable:
For our four-node, four-host MySQL Cluster, we will need to write four
configuration files, one per node/host.
Each data node or SQL node requires a
my.cnf file that provides two pieces
of information: a
connectstring telling the
node where to find the MGM node, and a line telling the MySQL server
on this host (the machine hosting the data node) to run in NDB mode.
The management node needs a config.ini
file telling it how many replicas to maintain, how much memory to
allocate for data and indexes on each data node, where to find the
data nodes, where to save data to disk on each data node, and where
to find any SQL nodes.
Configuring the Storage and SQL Nodes
The my.cnf file needed for the data nodes
is fairly simple. The configuration file should be located in the
/etc directory and can be edited using any text editor. (Create
the file if it does not exist.) For example:
shell> vi /etc/my.cnf
We show vi being used here
to create the file, but any text editor should work just as well.
For each data node and SQL node in our example setup,
my.cnf should look like this:
# Options for mysqld process:
[MYSQLD]
ndbcluster # run NDB engine
ndb-connectstring=192.168.0.10 # location of MGM node
# Options for ndbd process:
[MYSQL_CLUSTER]
ndb-connectstring=192.168.0.10 # location of MGM node
After entering the preceding information, save this file and exit the
text editor. Do this for the machines hosting data node “A”,
data node “B”, and the SQL node.
Important: Once you have
started a mysqld process
with the
ndbcluster and
ndb-connectstring parameters in the
[MYSQLD] in the
my.cnf
file as shown previously, you cannot execute any
CREATE TABLE or ALTER TABLE
statements without having actually started the cluster. Otherwise, these
statements will fail with an error. This is
by design.
Configuring the Management Node
The first step in configuring the MGM node is to create the directory in
which the configuration file can be found and then to create the file
itself. For example (running as
root):
shell> mkdir /var/lib/mysql-cluster
shell> cd /var/lib/mysql-cluster
shell> vi config.ini
For our representative setup, the
config.ini file should read as follows:
# Options affecting ndbd processes on all data nodes:
[NDBD DEFAULT]
NoOfReplicas=2 # Number of replicas
DataMemory=80M # How much memory to allocate for data storage
IndexMemory=18M # How much memory to allocate for index storage
# For DataMemory and IndexMemory, we have used the
# default values. Since the "world" database takes up
# only about 500KB, this should be more than enough for
# this example Cluster setup.
# TCP/IP options:
[TCP DEFAULT]
portnumber=2202 # This the default; however, you can use any
# port that is free for all the hosts in cluster
# Note: It is recommended beginning with MySQL 5.0 that
# you do not specify the portnumber at all and simply allow
# the default value to be used instead
# Management process options:
[NDB_MGMD]
hostname=192.168.0.10 # Hostname or IP address of MGM node
datadir=/var/lib/mysql-cluster # Directory for MGM node logfiles
# Options for data node "A":
[NDBD]
# (one [NDBD] section per data node)
hostname=192.168.0.30 # Hostname or IP address
datadir=/usr/local/mysql/data # Directory for this data node's datafiles
# Options for data node "B":
[NDBD]
hostname=192.168.0.40 # Hostname or IP address
datadir=/usr/local/mysql/data # Directory for this data node's datafiles
# SQL node options:
[MYSQLD]
hostname=192.168.0.20 # Hostname or IP address
# (additional mysqld connections can be
# specified for this node for various
# purposes such as running ndb_restore)
(Note: The
world database can be downloaded from
http://dev.mysql.com/doc/, where it can be found listed under “Examples.”)
After all the configuration files have been created and these minimal
options have been specified, you are ready to proceed with starting the
cluster and verifying that all processes are running. We discuss how
this is done in
Section 15.3.4, “Initial Startup”.
Note: The default port for
Cluster management nodes is 1186; the default port for data nodes is
2202. In MySQL 4.1, ports for data nodes are allocated sequentially
beginning with port 2202 and these ports must be available for the
cluster to use.
15.3.4. Initial Startup
Starting the cluster is not very difficult after it has been configured.
Each cluster node process must be started separately, and on the host
where it resides. Although it is possible to start the nodes in any
order, it is recommended that the management node be started first,
followed by the storage nodes, and then finally by any SQL nodes:
On the management host, issue the following command from the system
shell to start the MGM node process:
On each of the data node hosts, run this command to start the
ndbd process for the first time:
shell> ndbd --initial
Note that it is very important to use the
--initial parameter
only when starting
ndbd for the first
time, or when restarting after a backup/restore operation or a
configuration change. This is because the
--initial option causes the node to
delete any files created by earlier
ndbd
instances that are needed for recovery, including the recovery log
files.
If you used RPM files to install MySQL on the cluster host where the
SQL node is to reside, you can (and should) use the startup script
installed in
/etc/init.d to start the MySQL server
process on the SQL node. Note that you need to install the
-max server RPM
in addition to the Standard server RPM to run the
-max server binary.
If all has gone well, and the cluster has been set up correctly, the
cluster should now be operational. You can test this by invoking the
ndb_mgm management node client. The output should look
like that shown here, although you might see some slight differences in
the output depending upon the exact version of MySQL that you are using:
Note: If you are using an
older version of MySQL, you may see the SQL node referenced as
[mysqld(API)]. This reflects an older usage
that is now deprecated.
15.3.5. Loading Sample Data and Performing Queries
Working with data in MySQL Cluster is not much different from doing so
in MySQL without Cluster. There are two points to keep in mind:
For a table to be replicated in the cluster, it must use the
NDB Cluster storage engine. To specify
this, use the ENGINE=NDB or
ENGINE=NDBCLUSTER table option. You can
add this option when creating the table:
CREATE TABLE tbl_name ( ... ) ENGINE=NDBCLUSTER;
Alternatively, for an existing table that uses a different storage
engine, use ALTER TABLE to change the
table to use NDB Cluster:
ALTER TABLE tbl_name ENGINE=NDBCLUSTER;
Each NDB table
must
have a primary key. If no primary key is defined by the user when a
table is created, the NDB Cluster
storage engine automatically generates a hidden one. (Note:
This hidden key takes up space just as does any other table index.
It is not uncommon to encounter problems due to insufficient memory
for accommodating these automatically created indexes.)
If you are importing tables from an existing database using the output
of mysqldump, you can open
the SQL script in a text editor and add the ENGINE
option to any table creation statements, or replace any existing
ENGINE (or TYPE)
options. Suppose that you have the world
sample database on another MySQL server that does not support MySQL
Cluster, and you want to export the City
table:
shell> mysqldump --add-drop-table world City > city_table.sql
The resulting city_table.sql file will
contain this table creation statement (and the
INSERT statements necessary to import the
table data):
DROP TABLE IF EXISTS `City`;
CREATE TABLE `City` (
`ID` int(11) NOT NULL auto_increment,
`Name` char(35) NOT NULL default '',
`CountryCode` char(3) NOT NULL default '',
`District` char(20) NOT NULL default '',
`Population` int(11) NOT NULL default '0',
PRIMARY KEY (`ID`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1;
INSERT INTO `City` VALUES (1,'Kabul','AFG','Kabol',1780000);
INSERT INTO `City` VALUES (2,'Qandahar','AFG','Qandahar',237500);
INSERT INTO `City` VALUES (3,'Herat','AFG','Herat',186800);
(remaining INSERT statements omitted)
You will need to make sure that MySQL uses the NDB storage engine for
this table. There are two ways that this can be accomplished. One of
these is to modify the table definition
before importing it into the
Cluster database. Using the City table as
an example, modify the ENGINE option of the
definition as follows:
DROP TABLE IF EXISTS `City`;
CREATE TABLE `City` (
`ID` int(11) NOT NULL auto_increment,
`Name` char(35) NOT NULL default '',
`CountryCode` char(3) NOT NULL default '',
`District` char(20) NOT NULL default '',
`Population` int(11) NOT NULL default '0',
PRIMARY KEY (`ID`)
) ENGINE=NDBCLUSTER DEFAULT CHARSET=latin1;
INSERT INTO `City` VALUES (1,'Kabul','AFG','Kabol',1780000);
INSERT INTO `City` VALUES (2,'Qandahar','AFG','Qandahar',237500);
INSERT INTO `City` VALUES (3,'Herat','AFG','Herat',186800);
(remaining INSERT statements omitted)
This must be done for the definition of each table that is to be part of
the clustered database. The easiest way to accomplish this is to do a
search-and-replace on the file that contains the definitions and replace
all instances of
TYPE=engine_name
or
ENGINE=engine_name
with ENGINE=NDBCLUSTER. If you do not want
to modify the file, you can use the unmodified file to create the
tables, and then use ALTER TABLE to change
their storage engine. The particulars are given later in this section.
Assuming that you have already created a database named
world on the SQL node of the cluster, you
can then use the mysql
command-line client to read city_table.sql,
and create and populate the corresponding table in the usual manner:
shell> mysql world < city_table.sql
It is very important to keep in mind that the preceding command must be
executed on the host where the SQL node is running (in this case, on the
machine with the IP address
192.168.0.20).
To create a copy of the entire world
database on the SQL node, use mysqldump
on the non-cluster server to export the database to a file named
world.sql; for example, in the
/tmp directory. Then modify the table
definitions as just described and import the file into the SQL node of
the cluster like this:
shell> mysql world < /tmp/world.sql
If you save the file to a different location, adjust the preceding
instructions accordingly.
It is important to note that NDB Cluster in
MySQL 4.1 does not support autodiscovery of databases. (See
Section 15.9, “Known Limitations of MySQL Cluster”.) This means
that, once the world database and its
tables have been created on one data node, you need to issue the
CREATE DATABASE world statement followed by
FLUSH TABLES on each SQL node in the cluster. This will cause the
node to recognize the database and read its table definitions.
Running SELECT queries on the SQL node is
no different from running them on any other instance of a MySQL server.
To run queries from the command line, you first need to log in to the
MySQL Monitor in the usual way (specify the
root password at the
Enter password: prompt):
shell> mysql -u root -p
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 1 to server version: 4.1.21
Type 'help;' or '\h' for help. Type '\c' to clear the buffer.
mysql>
We simply use the MySQL server's root
account and assume that you have followed the standard security
precautions for installing a MySQL server, including setting a strong
root password. For more information, see
Section 2.10.3, “Securing the Initial MySQL Accounts”.
It is worth taking into account that Cluster nodes do not make use of
the MySQL privilege system when accessing one another. Setting or
changing MySQL user accounts (including the
root account) effects only applications
that access the SQL node, not interaction between nodes.
If you did not modify the ENGINE clauses in
the table definitions prior to importing the SQL script, you should run
the following statements at this point:
mysql> USE world;
mysql> ALTER TABLE City ENGINE=NDBCLUSTER;
mysql> ALTER TABLE Country ENGINE=NDBCLUSTER;
mysql> ALTER TABLE CountryLanguage ENGINE=NDBCLUSTER;
Selecting a database and running a SELECT
query against a table in that database is also accomplished in the usual
manner, as is exiting the MySQL Monitor:
mysql> USE world;
mysql> SELECT Name, Population FROM City ORDER BY Population DESC LIMIT 5;
+-----------+------------+
| Name | Population |
+-----------+------------+
| Bombay | 10500000 |
| Seoul | 9981619 |
| São Paulo | 9968485 |
| Shanghai | 9696300 |
| Jakarta | 9604900 |
+-----------+------------+
5 rows in set (0.34 sec)
mysql> \q
Bye
shell>
Applications that use MySQL can employ standard APIs to access NDB
tables. It is important to remember that your application must access
the SQL node, and not the MGM or data nodes. This brief example shows
how we might execute the
SELECT statement just shown by using PHP
5's
mysqli extension running on a Web server
elsewhere on the network:
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
"http://www.w3.org/TR/html4/loose.dtd">
<html>
<head>
<meta http-equiv="Content-Type"
content="text/html; charset=iso-8859-1">
<title>SIMPLE mysqli SELECT</title>
</head>
<body>
<?php
# connect to SQL node:
$link = new mysqli('192.168.0.20', 'root', 'root_password', 'world');
# parameters for mysqli constructor are:
# host, user, password, database
if( mysqli_connect_errno() )
die("Connect failed: " . mysqli_connect_error());
$query = "SELECT Name, Population
FROM City
ORDER BY Population DESC
LIMIT 5";
# if no errors...
if( $result = $link->query($query) )
{
?>
<table border="1" width="40%" cellpadding="4" cellspacing ="1">
<tbody>
<tr>
<th width="10%">City</th>
<th>Population</th>
</tr>
<?
# then display the results...
while($row = $result->fetch_object())
printf(<tr>\n <td align=\"center\">%s</td><td>%d</td>\n</tr>\n",
$row->Name, $row->Population);
?>
</tbody
</table>
<?
# ...and verify the number of rows that were retrieved
printf("<p>Affected rows: %d</p>\n", $link->affected_rows);
}
else
# otherwise, tell us what went wrong
echo mysqli_error();
# free the result set and the mysqli connection object
$result->close();
$link->close();
?>
</body>
</html>
We assume that the process running on the Web server can reach the IP
address of the SQL node.
In a similar fashion, you can use the MySQL C API, Perl-DBI,
Python-mysql, or MySQL AB's own Connectors to perform the tasks of data
definition and manipulation just as you would normally with MySQL.
15.3.6. Safe Shutdown and Restart
To shut down the cluster, enter the following command in a shell on the
machine hosting the MGM node:
shell> ndb_mgm -e shutdown
The -e option here is used to pass a command
to the ndb_mgm client from
the shell. See
Section 4.3.1, “Using Options on the Command Line”. The command
causes the
ndb_mgm,
ndb_mgmd, and any
ndbd processes to
terminate gracefully. Any SQL nodes can be terminated using
mysqladmin shutdown and other means.
To restart the cluster, run these commands:
On the management host (192.168.0.10 in
our example setup):
A MySQL server that is part of a MySQL Cluster differs in only one respect
from a normal (non-clustered) MySQL server, in that it employs the
NDB Cluster storage engine. This engine is also referred to
simply as NDB, and the two forms of the
name are synonymous.
To avoid unnecessary allocation of resources, the server is configured by
default with the NDB storage engine
disabled. To enable NDB, you must modify
the server's my.cnf configuration file, or
start the server with the --ndbcluster
option.
The MySQL server is a part of the cluster, so it also must know how to
access an MGM node to obtain the cluster configuration data. The default
behavior is to look for the MGM node on
localhost. However, should you need to
specify that its location is elsewhere, this can be done in
my.cnf or on the MySQL server command line.
Before the NDB storage engine can be used,
at least one MGM node must be operational, as well as any desired data
nodes.
15.4.1. Building MySQL Cluster from Source Code
NDB, the Cluster storage engine, is
available in binary distributions for Linux, Mac OS X, and Solaris. We
are working to make Cluster run on all operating systems supported by
MySQL, including Windows.
If you choose to build from a source tarball or the MySQL 4.1 BitKeeper
tree, be sure to use the
--with-ndbcluster option when running
configure. You can also
use the
BUILD/compile-pentium-max
build script. Note that this script includes OpenSSL, so you must either
have or obtain OpenSSL to build successfully, or else modify
compile-pentium-max to
exclude this requirement. Of course, you can also just follow the
standard instructions for compiling your own binaries, and then perform
the usual tests and installation procedure. See
Section 2.9.3, “Installing from the Development Source Tree”.
15.4.2. Installing the Software
In the next few sections, we assume that you are already familiar with
installing MySQL, and here we cover only the differences between
configuring MySQL Cluster and configuring MySQL without clustering. (See
Chapter 2, Installing and Upgrading MySQL, if you require
more information about the latter.)
You will find Cluster configuration easiest if you have already have all
management and data nodes running first; this is likely to be the most
time-consuming part of the configuration. Editing the
my.cnf file is fairly straightforward, and this section will
cover only any differences from configuring MySQL without clustering.
15.4.3. Quick Test Setup of MySQL Cluster
To familiarize you with the basics, we will describe the simplest
possible configuration for a functional MySQL Cluster. After this, you
should be able to design your desired setup from the information
provided in the other relevant sections of this chapter.
First, you need to create a configuration directory such as
/var/lib/mysql-cluster, by executing the
following command as the system root user:
shell> mkdir /var/lib/mysql-cluster
In this directory, create a file named
config.ini that contains the following
information. Substitute appropriate values for
HostName and DataDir
as necessary for your system.
# file "config.ini" - showing minimal setup consisting of 1 data node,
# 1 management server, and 3 MySQL servers.
# The empty default sections are not required, and are shown only for
# the sake of completeness.
# Data nodes must provide a hostname but MySQL Servers are not required
# to do so.
# If you don't know the hostname for your machine, use localhost.
# The DataDir parameter also has a default value, but it is recommended to
# set it explicitly.
# Note: DB, API, and MGM are aliases for NDBD, MYSQLD, and NDB_MGMD
# respectively. DB and API are deprecated and should not be used in new
# installations.
[NDBD DEFAULT]
NoOfReplicas= 1
[MYSQLD DEFAULT]
[NDB_MGMD DEFAULT]
[TCP DEFAULT]
[NDB_MGMD]
HostName= myhost.example.com
[NDBD]
HostName= myhost.example.com
DataDir= /var/lib/mysql-cluster
[MYSQLD]
[MYSQLD]
[MYSQLD]
You can now start the ndb_mgmd
management server. By default, it atttempts to read the
config.ini file in its current working
directory, so change location into the directory where the file is
located and then invoke ndb_mgmd:
shell> cd /var/lib/mysql-cluster
shell> ndb_mgmd
Then start a single data node by running
ndbd. When starting
ndbd
for a given data node for the very first time, you should use the
--initial option as shown here:
shell> ndbd --initial
For subsequent ndbd
starts, you will generally want to omit
the
--initial option:
shell> ndbd
The reason for omitting --initial on
subsequent restarts is that this option causes
ndbd to delete and re-create all existing data and log
files (as well as all table metadata) for this data node. One exception
to this rule about not using --initial
except for the first ndbd
invocation is that you use it when restarting the cluster and restoring
from backup after adding new data nodes.
By default, ndbd looks for
the management server at localhost on port
1186. (Prior to MySQL 4.1.8, the default port was 2200.)
Note: If you have installed
MySQL from a binary tarball, you will need to specify the path of the
ndb_mgmd and ndbd
servers explicitly. (Normally, these will be found in
/usr/local/mysql/bin.)
Finally, change location to the MySQL data directory (usually
/var/lib/mysql or
/usr/local/mysql/data), and make sure that
the my.cnf file contains the option
necessary to enable the NDB storage engine:
[mysqld]
ndbcluster
You can now start the MySQL server as usual:
shell> mysqld_safe --user=mysql &
Wait a moment to make sure the MySQL server is running properly. If you
see the notice mysql ended, check the
server's .err file to find out what went
wrong.
If all has gone well so far, you now can start using the cluster.
Connect to the server and verify that the
NDBCLUSTER storage engine is enabled:
shell> mysql
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 1 to server version: 4.1.21-Max
Type 'help;' or '\h' for help. Type '\c' to clear the buffer.
mysql> SHOW ENGINES\G
...
*************************** 12. row ***************************
Engine: NDBCLUSTER
Support: YES
Comment: Clustered, fault-tolerant, memory-based tables
*************************** 13. row ***************************
Engine: NDB
Support: YES
Comment: Alias for NDBCLUSTER
...
The row numbers shown in the preceding example output may be different
from those shown on your system, depending upon how your server is
configured.
Try to create an NDBCLUSTER table:
shell> mysql
mysql> USE test;
Database changed
mysql> CREATE TABLE ctest (i INT) ENGINE=NDBCLUSTER;
Query OK, 0 rows affected (0.09 sec)
mysql> SHOW CREATE TABLE ctest \G
*************************** 1. row ***************************
Table: ctest
Create Table: CREATE TABLE `ctest` (
`i` int(11) default NULL
) ENGINE=ndbcluster DEFAULT CHARSET=latin1
1 row in set (0.00 sec)
To check that your nodes were set up properly, start the management
client:
shell> ndb_mgm
Use the SHOW command from
within the management client to obtain a report on the cluster's status:
NDB> SHOW
Cluster Configuration
---------------------
[ndbd(NDB)] 1 node(s)
id=2 @127.0.0.1 (Version: 3.5.3, Nodegroup: 0, Master)
[ndb_mgmd(MGM)] 1 node(s)
id=1 @127.0.0.1 (Version: 3.5.3)
[mysqld(API)] 3 node(s)
id=3 @127.0.0.1 (Version: 3.5.3)
id=4 (not connected, accepting connect from any host)
id=5 (not connected, accepting connect from any host)
At this point, you have successfully set up a working MySQL Cluster. You
can now store data in the cluster by using any table created with
ENGINE=NDBCLUSTER or its alias ENGINE=NDB.
Configuring MySQL Cluster requires working with two files:
my.cnf: Specifies options for all
MySQL Cluster executables. This file, with which you should be
familiar with from previous work with MySQL, must be accessible by
each executable running in the cluster.
config.ini: This file is read only by
the MySQL Cluster management server, which then distributes the
information contained therein to all processes participating in the
cluster.
config.ini contains a description of
each node involved in the cluster. This includes configuration
parameters for data nodes and configuration parameters for
connections between all nodes in the cluster.
We are continuously making improvements in Cluster configuration and
attempting to simplify this process. Although we strive to maintain
backward compatibility, there may be times when introduce an
incompatible change. In such cases we will try to let Cluster users know
in advance if a change is not backward compatible. If you find such a
change and we have not documented it, please report it in the MySQL bugs
database using the instructions given in
Section 1.8, “How to Report Bugs or Problems”.
15.4.4.1. Example Configuration for a MySQL Cluster
To support MySQL Cluster, you will need to update
my.cnf as shown in the following
example. Note that the options shown here should not be confused
with those that are used in config.ini
files. You may also specify these parameters on the command line
when invoking the executables.
From version 4.1.8 some simplifications in
my.cnf were made, including new sections
for ndbcluster executables. However,
these should not be confused with those occurring in
config.ini files. As always, you may
specify these parameters when invoking those executables from the
command line.
# my.cnf
# example additions to my.cnf for MySQL Cluster
# (valid from 4.1.8)
# enable ndbcluster storage engine, and provide connectstring for
# management server host (default port is 1186)
[mysqld]
ndbcluster
ndb-connectstring=ndb_mgmd.mysql.com
# provide connectstring for management server host (default port: 1186)
[ndbd]
connect-string=ndb_mgmd.mysql.com
# provide connectstring for management server host (default port: 1186)
[ndb_mgm]
connect-string=ndb_mgmd.mysql.com
# provide location of cluster configuration file
[ndb_mgmd]
config-file=/etc/config.ini
# my.cnf
# example additions to my.cnf for MySQL Cluster
# (will work on all versions)
# enable ndbcluster storage engine, and provide connectstring for management
# server host to the default port 1186
[mysqld]
ndbcluster
ndb-connectstring=ndb_mgmd.mysql.com:1186
Important: Once you have
started a mysqld
process with the
ndbcluster and
ndb-connectstring parameters in the
[MYSQLD] in the
my.cnf
file as shown previously, you cannot execute any
CREATE TABLE or
ALTER TABLE statements without having actually started the
cluster. Otherwise, these statements will fail with an error.
This is by design.
Starting with MySQL 4.1.8, you may also use a separate
[mysql_cluster] section in the cluster
my.cnf for settings to be read and used
by all executables:
The configuration file is named
config.ini by default. It is read by
ndb_mgmd at startup and
can be placed anywhere. Its location and name are specified by using
--config-file=path_name
on the ndb_mgmd command
line. If the configuration file is not specified,
ndb_mgmd by default
tries to read a file named config.ini
located in the current working directory.
Currently, the configuration file is in INI format, which consists of
sections preceded by section headings (surrounded by square
brackets), followed by the appropriate parameter names and values.
One deviation from the standard INI format is that the parameter
name and value can be separated by a colon (‘:’)
as well as the equals sign (‘=’).
Another deviation is that sections are not uniquely identified by
section name. Instead, unique sections (such as two different nodes
of the same type) are identified by a unique ID specified as a
parameter within the section.
Default values are defined for most parameters, and can also be
specified in config.ini. To create a
default value section, simply add the word
DEFAULT to the section name. For example,
an [NDBD] section contains parameters
that apply to a particular data node, whereas an
[NDBD DEFAULT] section contains parameters that apply to all
data nodes. Suppose that all data nodes should use the same data
memory size. To configure them all, create an
[NDBD DEFAULT] section that contains a
DataMemory line to specify the data
memory size.
At a minimum, the configuration file must define the computers and
nodes involved in the cluster and on which computers these nodes are
located. An example of a simple configuration file for a cluster
consisting of one management server, two data nodes and two MySQL
servers is shown here:
# file "config.ini" - 2 data nodes and 2 SQL nodes
# This file is placed in the startup directory of ndb_mgmd (the
# management server)
# The first MySQL Server can be started from any host. The second
# can be started only on the host mysqld_5.mysql.com
[NDBD DEFAULT]
NoOfReplicas= 2
DataDir= /var/lib/mysql-cluster
[NDB_MGMD]
Hostname= ndb_mgmd.mysql.com
DataDir= /var/lib/mysql-cluster
[NDBD]
HostName= ndbd_2.mysql.com
[NDBD]
HostName= ndbd_3.mysql.com
[MYSQLD]
[MYSQLD]
HostName= mysqld_5.mysql.com
Note that each node has its own section in the
config.ini. For instance, this cluster
has two data nodes, so the preceding configuration file contains two
[NDBD] sections defining these nodes.
There are six different sections in that you can use in the
config.ini configuration file:
[COMPUTER]: Defines the cluster
hosts.
[NDBD]: Defines the cluster's data
nodes.
[MYSQLD]: Defines the cluster's MySQL
server nodes.
[MGM] or
[NDB_MGMD]: Defines the cluster's management server node.
[TCP]: Defines TCP/IP connections
between nodes in the cluster, with TCP/IP being the default
connection protocol.
[SHM]: Defines shared-memory
connections between nodes. Prior to MySQL 4.1.9, this type of
connection was available only in binaries that were built using
the --with-ndb-shm option. Beginning
with MySQL 4.1.9-max, it is enabled by default, but should still
be considered experimental.
You can define DEFAULT values for each
section. As of MySQL 4.1.5, all parameter names are
case-insensitive, which differs from parameters specified in
my.cnf or my.ini
files.
15.4.4.2. The MySQL Cluster connectstring
With the exception of the MySQL Cluster management server (ndb_mgmd),
each node that is part of a MySQL Cluster requires a
connectstring
that points to the management server's location. This connectstring is
used in establishing a connection to the management server as well
as in performing other tasks depending on the node's role in the
cluster. The syntax for a connectstring is as follows:
node_id is an integer larger than 1 which
identifies a node in config.ini.
host_name is a string
representing a valid Internet host name or IP address.
port_num is an integer
referring to a TCP/IP port number.
example 1 (long): "nodeid=2,myhost1:1100,myhost2:1100,192.168.0.3:1200"
example 2 (short): "myhost1"
All nodes will use localhost:1186 as the
default connectstring value if none is provided. If
port_num is omitted from the
connectstring, the default port is 1186. (Note:
Prior to MySQL 4.1.8, the default port was 2200.) This port should
always be available on the network because it has been assigned by
IANA for this purpose (see
http://www.iana.org/assignments/port-numbers
for details).
By listing multiple
<host-specification> values, it is
possible to designate several redundant management servers. A
cluster node will attempt to contact successive management servers
on each host in the order specified, until a successful connection
has been established.
There are a number of different ways to specify the connectstring:
Each executable has its own command-line option which enables
specifying the management server at startup. (See the
documentation for the respective executable.)
Beginning with MySQL 4.1.8, it is also possible to set the
connectstring for all nodes in the cluster at once by placing it
in a [mysql_cluster] section in the
management server's my.cnf
file.
For backward compatibility, two other options are available, using
the same syntax:
Set the NDB_CONNECTSTRING
environment variable to contain the connectstring.
Write the connectstring for each executable into a text file
named Ndb.cfg and place this
file in the executable's startup directory.
However, these are now deprecated and should not be used for new
installations.
The recommended method for specifying the connectstring is to set it
on the command line or in the
my.cnf file for each executable.
15.4.4.3. Defining the Computers Making up a MySQL Cluster
The [COMPUTER] section has no real
significance other than serving as a way to avoid the need of
defining host names for each node in the system. All parameters
mentioned here are required.
Id
This is an integer value, used to refer to the host computer
elsewhere in the configuration file.
HostName
This is the computer's hostname or IP address.
15.4.4.4. Defining the MySQL Cluster Management Server
The [NDB_MGMD] section is used to
configure the behavior of the management server.
[MGM] can be used as an alias; the two
section names are equivalent. All parameters in the following list
are optional and assume their default values if omitted.
Note: If neither the
ExecuteOnComputer nor the
HostName parameter is present, the
default value localhost will be assumed
for both.
Id
Each node in the cluster has a unique identity, which is
represented by an integer value in the range 1 to 63 inclusive.
This ID is used by all internal cluster messages for addressing
the node.
ExecuteOnComputer
This refers to one of the computers defined in the
[COMPUTER] section.
PortNumber
This is the port number on which the management server listens for
configuration requests and management commands.
LogDestination
This parameter specifies where to send cluster logging
information. There are three options in this regard:
CONSOLE, SYSLOG,
and
FILE:
CONSOLE outputs the log to
stdout:
CONSOLE
SYSLOG sends the log to a
syslog facility, possible values
being one of auth,
authpriv,
cron,
daemon, ftp,
kern, lpr,
mail, news,
syslog,
user,
uucp,
local0,
local1,
local2,
local3,
local4,
local5,
local6, or local7.
Note: Not every
facility is necessarily supported by every operating system.
SYSLOG:facility=syslog
FILE pipes the cluster log output
to a regular file on the same machine. The following values
can be specified:
filename: The name of the
logfile.
maxsize: The maximum size (in
bytes) to which the file can grow before logging rolls
over to a new file. When this occurs, the old logfile is
renamed by appending
.N to the
filename, where N
is the next number not yet used with this name.
The default value for the FILE
parameter is
FILE:filename=ndb_node_id_cluster.log,maxsize=1000000,maxfiles=6,
where node_id is
the ID of the node.
ArbitrationRank
This parameter is used to define which nodes can act as
arbitrators. Only MGM nodes and SQL nodes can be arbitrators.
ArbitrationRank can take one of the following values:
0: The node will never be used as
an arbitrator.
1: The node has high priority;
that is, it will be preferred as an arbitrator over
low-priority nodes.
2: Indicates a low-priority node
which be used as an arbitrator only if a node with a higher
priority is not available for that purpose.
Normally, the management server should be configured as an
arbitrator by setting its
ArbitrationRank to 1 (the default
value) and that of all SQL nodes to 0.
ArbitrationDelay
An integer value which causes the management server's responses to
arbitration requests to be delayed by that number of
milliseconds. By default, this value is 0; it is normally not
necessary to change it.
DataDir
This specifies the directory where output files from the
management server will be placed. These files include cluster
log files, process output files, and the daemon's process ID
(PID) file. (For log files, this location can be overridden by
setting the FILE
parameter for LogDestination as
discussed previously in this section.)
15.4.4.5. Defining MySQL Cluster Data Nodes
The [NDBD] section is used to configure
the behavior of the cluster's data nodes. There are many parameters
which control buffer sizes, pool sizes, timeouts, and so forth. The
only mandatory parameters are:
Either ExecuteOnComputer or
HostName.
The parameter NoOfReplicas
These mandatory parameters must be defined in the
[NDBD DEFAULT] section.
Most data node parameters are set in the [NDBD
DEFAULT] section. Only those parameters explicitly stated as
being able to set local values are allowed to be changed in the
[NDBD] section.
HostName, Id
and
ExecuteOnComputer
must
be defined in the local [NDBD] section.
Identifying Data Nodes
The Id value (that is, the data node
identifier) can be allocated on the command line when the node is
started or in the configuration file.
For each parameter it is possible to use K,
M, or G as a
suffix to indicate units of 1024, 1024×1024, or 1024×1024×1024. (For
example,
100K means 100 × 1024 = 102400.)
Parameter names and values are currently case-sensitive.
Id
This is the node ID used as the address of the node for all
cluster internal messages. This is an integer in the range 1 to
63 inclusive. Each node in the cluster must have a unique
identity.
ExecuteOnComputer
This refers to one of the computers (hosts) defined in the
COMPUTER section.
HostName
Specifying this parameter has an effect similar to specifying
ExecuteOnComputer. It defines the hostname of the
computer on which the storage node is to reside. To specify a
hostname other than
localhost, either this parameter or
ExecuteOnComputer is required.
ServerPort
(OBSOLETE)
Each node in the cluster uses a port to connect to other nodes.
This port is used also for non-TCP transporters in the
connection setup phase. The default port is allocated
dynamically in such a way as to ensure that no two nodes on the
same computer receive the same port number, so it should not
normally be necessary to specify a value for this parameter.
NoOfReplicas
This global parameter can be set only in the
[NDBD DEFAULT] section, and defines
the number of replicas for each table stored in the cluster.
This parameter also specifies the size of node groups. A node
group is a set of nodes all storing the same information.
Node groups are formed implicitly. The first node group is formed
by the set of data nodes with the lowest node IDs, the next node
group by the set of the next lowest node identities, and so on.
By way of example, assume that we have 4 data nodes and that
NoOfReplicas
is set to 2. The four data nodes have node IDs 2, 3, 4 and 5. Then
the first node group is formed from nodes 2 and 3, and the
second node group by nodes 4 and 5. It is important to configure
the cluster in such a manner that nodes in the same node groups
are not placed on the same computer because a single hardware
failure would cause the entire cluster to crash.
If no node IDs are provided, the order of the data nodes will be
the determining factor for the node group. Whether or not
explicit assignments are made, they can be viewed in the output
of the management client's
SHOW statement.
There is no default value for
NoOfReplicas; the maximum possible
value is 4.
DataDir
This parameter specifies the directory where trace files, log
files, pid files and error logs are placed.
FileSystemPath
This parameter specifies the directory where all files created for
metadata, REDO logs, UNDO logs and data files are placed. The
default is the directory specified by
DataDir.
Note: This directory
must exist before the ndbd
process is initiated.
The recommended directory hierarchy for MySQL Cluster includes
/var/lib/mysql-cluster, under which a directory for the
node's filesystem is created. The name of this subdirectory
contains the node ID. For example, if the node ID is 2, this
subdirectory is named ndb_2_fs.
BackupDataDir
This parameter specifies the directory in which backups are
placed. If omitted, the default backup location is the directory
named BACKUP under the location
specified by the
FileSystemPath parameter. (See
above.)
Data Memory, Index Memory, and String
Memory
DataMemory and
IndexMemory are
[NDBD]
parameters specifying the size of memory segments used to store the
actual records and their indexes. In setting values for these, it is
important to understand how
DataMemory and
IndexMemory are used, as they usually
need to be updated to reflect actual usage by the cluster:
DataMemory
This parameter defines the amount of space (in bytes) available
for storing database records. The entire amount specified by
this value is allocated in memory, so it is extremely important
that the machine has sufficient physical memory to accommodate
it.
The memory allocated by DataMemory is
used to store both the actual records and indexes. Each record
is currently of fixed size. (Even
VARCHAR columns are stored as
fixed-width columns.) There is a 16-byte overhead on each
record; an additional amount for each record is incurred because
it is stored in a 32KB page with 128 byte page overhead (see
below). There is also a small amount wasted per page due to the
fact that each record is stored in only one page. The maximum
record size is currently 8052 bytes.
The memory space defined by DataMemory
is also used to store ordered indexes, which use about 10 bytes
per record. Each table row is represented in the ordered index.
A common error among users is to assume that all indexes are
stored in the memory allocated by
IndexMemory, but this is not the
case: Only primary key and unique hash indexes use this memory;
ordered indexes use the memory allocated by
DataMemory. However, creating a
primary key or unique hash index also creates an ordered index
on the same keys, unless you specify USING
HASH in the index creation statement. This can be
verified by running ndb_desc -d
db_nametable_name
in the management client.
The memory space allocated by
DataMemory consists of 32KB pages,
which are allocated to table fragments. Each table is normally
partitioned into the same number of fragments as there are data
nodes in the cluster. Thus, for each node, there are the same
number of fragments as are set in
NoOfReplicas.
Once a page has been allocated, it is currently not possible to
return it to the pool of free pages, except by deleting the
table. (This also means that
DataMemory pages, once allocated to a
given table, cannot be used by other tables.) Performing a node
recovery also compresses the partition because all records are
inserted into empty partitions from other live nodes.
The DataMemory memory space also
contains UNDO information: For each update, a copy of the
unaltered record is allocated in the
DataMemory. There is also a reference
to each copy in the ordered table indexes. Unique hash indexes
are updated only when the unique index columns are updated, in
which case a new entry in the index table is inserted and the
old entry is deleted upon commit. For this reason, it is also
necessary to allocate enough memory to handle the largest
transactions performed by applications using the cluster. In any
case, performing a few large transactions holds no advantage
over using many smaller ones, for the following reasons:
Large transactions are not any faster than smaller ones
Large transactions increase the number of operations that are
lost and must be repeated in event of transaction failure
Large transactions use more memory
The default value for DataMemory is
80MB; the minimum is 1MB. There is no maximum size, but in
reality the maximum size has to be adapted so that the process
does not start swapping when the limit is reached. This limit is
determined by the amount of physical RAM available on the
machine and by the amount of memory that the operating system
may commit to any one process. 32-bit operating systems are
generally limited to 2–4GB per process; 64-bit operating systems
can use more. For large databases, it may be preferable to use a
64-bit operating system for this reason. In addition, it is also
possible to run more than one ndbd
process per machine, and this may prove advantageous on machines
with multiple CPUs.
IndexMemory
This parameter controls the amount of storage used for hash
indexes in MySQL Cluster. Hash indexes are always used for
primary key indexes, unique indexes, and unique constraints.
Note that when defining a primary key and a unique index, two
indexes will be created, one of which is a hash index used for
all tuple accesses as well as lock handling. It is also used to
enforce unique constraints.
The size of the hash index is 25 bytes per record, plus the size
of the