Comment by Slawomir on Splitting large database into smaller ones
Thank you. Do you agree that it's not possible to have one common namespace (database) hosting all the stored procedures, then be able to invoke it as shared.sp_get_info() in the context of another...
View ArticleComment by Slawomir on Splitting large database into smaller ones
We think so; we insert millions of rows a day and concurrency is insane. Scaling up (larger instances) will eventually hit a hard limit. We are in the same boat as these guys were back in 2012:...
View ArticleComment by Slawomir on Managing 100s of similar databases
More info added based on your feedback. THANKS!
View ArticleComment by Slawomir on Managing 100s of similar databases
The reason isn't just "good" - I could fight that one - it's mandated by the regulations.
View ArticleComment by Slawomir on Point in time recovery with mariabackup
right, that was my understanding too reading the docs and it's quite inconvenient... yet another tool (mysqlbinlog) thrown into the mix and the opportunity for an error
View ArticleComment by Slawomir on Minimizing MySQL downtime during OS update?
In #4, when the FLUSH... command finishes we'd issue the CHANGE MASTER TO...
View ArticleComment by Slawomir on Under load, the following job-queue design does not...
Jobs take sub-second to execute. There is an alarm whenever/if jobs were to take more than 3 std dev. The alarm works. There is about 50 jobs per second. The job_queue table has 4 columns,...
View ArticleComment by Slawomir on Under load, the following job-queue design does not...
Jobs may fail for various reasons, hence the status column. Queuing is a safety measure to ensure that if there is a backend processing problem, the queue will grow but the backend will have time to...
View ArticleComment by Slawomir on Under load, the following job-queue design does not...
I have rewrote my code to not rely on row_count(), just always select whatever UPDATE changed and deal with the results in the client. All previous problems got resolved.
View ArticleComment by Slawomir on Which FLUSH command before taking EBS volume snapshot?
It's too slow and too heavy on local resources (CPU, disk) compared to a volume snapshot. I recommend this article as it's universally applicable: engineering.shopify.com/blogs/engineering/…
View Articleguideline for calling optimize table after alter table?
Should I be calling 'optimize table' when changing datatypes of columns, adding indexes or dropping indexes, assuming a table with millions of rows, heavily used, both for reading and writing...
View ArticleMirror data on local SSD storage to conserve EBS I/O?
I have an AWS instance with local (ephemeral) SSD storage backed by a permanent EBS storage. Is it possible to configure the local storage to act as a mirror/cache for row data in MySQL? I do not wish...
View ArticleTrying to fit indexes in memory: partitioning vs compression or neither?
I have a "transaction" table that grows at about 150k rows-per-day rate. 97% of user aggregate queries refer to the last 3 months of data and must run very efficiently. Trimming older transaction data...
View ArticlePROCEDURE ANALYSE suggests to turn timestamp into CHAR(19)
Using Procedure Analyse() under MySQL 5.6.30 on a table with about 4 million rows makes the following recommendations:change timestamp to char(19)tinyint to ENUM (I noticed it's overzealous with ENUMS...
View ArticleEfficiency of foreign keys
If I have a table A which has a composite (two column) foreign key on another table B, should I:a) create a two-column key in B to match the foreign key in A?b) or, MySQL automatically creates the...
View ArticlePossible to have a statement in stored procedure outside of current transaction?
During a series of statements in my stored procedure I wish to log an entry by inserting a record. That record needs to survive even if the overall SP ends up rolling back.Is this possible?
View Articlepreferred order of columns in the key
If I have a table like so:table T ( account_id bigint, event_date datetime ... );and later make queries like so:select ...from Twhere account_id = 'ABC' and event_date >= '2016-01-01'Does the order...
View ArticleMySQL overhead of calling a Stored Procedure
Within Java, we access MySQL (v5.5.40) via JDBC -- a prepare statement, setting parameters then calling the stored procedure. It appears that every execute() has the overhead of 4 meta-calls (per MySQL...
View ArticleAcceptable deadlock rate and how to diagnose them
Our system is doing about 50 inserts per second. During ~2 hour period we are seeing up to 3 of "Deadlock found when trying to get lock; try restarting transaction" conditions. Yes, the client restarts...
View ArticleSimple conditional update causing a deadlock hints at MySQL bug
I have a very simple "inventory management" update that under load causes InnoDB to deadlock and rollback one of the threads. There doesn't appear to be any ordering issue here, so I'm at a loss how to...
View ArticleAnswer by Slawomir for Getting around MASTER_HOST length limit in 'CHANGE...
The way I resolved it is I clicked on Modify in AWS/RDS console and shortened the DB Cluster Identifier so the resulting cluster endpoint fit in 60 characters.
View ArticleClik here to view.
Getting around MASTER_HOST length limit in 'CHANGE MASTER TO...'?
I'm trying to configure external slave connecting to Aurora RDS. To my delight, I run into a silly problem where the Aurora cluster name - assigned by AWS - exceeds permissible MASTER_HOST limit of 60...
View ArticleReason for multiplicity of character/collation settings in MySQL?
The last 48 hours or so I spend on trying to determine the cause for sporadic crashes of my MySQL (Aurora) DB. Turns out they were attributed to my mixing of database, table, column and stored...
View ArticleAnswer by Slawomir for What does "Latest Restore Time" says in AWS RDS(Mysql)...
You can restore to any point in time during your backup retention period. To determine the latest restorable time for a DB instance, use the AWS CLI describe-db-instances command and look at the value...
View Articlerolling-up consecutive duplicates
Is there a way to "roll up" duplicate records such as time-series data in MySQL? I'm currently doing this in the application code after retrieving full time-series but the transmission of mostly...
View ArticleMySQL- reducing memory footprint
Our InnoDB/Mysql5.6 database is growing at a fairly steady pace of doubling in size every year, currently at 22Gb. This is sales data; we have a number of merchants each generating millions of rows,...
View ArticleSecuring LOAD DATA LOCAL
I need to periodically import significant number of rows from a CSV file. I'd like to be able to open up the least amount of access to the user (automation script) for this job.Is it possible to limit...
View ArticleAnswer by Slawomir for Changing location of binlog prevents starting up
I finally figured it out: MariaDB keeps an "index" file, fortunately ASCII, that you need to edit: mariadb-bin.index located in the binlog directory (or wherever your log_bin_index configuration points...
View ArticleFinding indexes that aren't used often or at all
Is there a facility or a recommended practice for determining which indexes are being used and which ones can be "purged"? At the minimum, I'm looking for a log file that over time can be analyzed for...
View ArticleAnswer by Slawomir for Under load, the following job-queue design does not...
I rewrote the logic such that the SELECT is run unconditionally, ie I got rid of the row_count() > 0 check and handle results in the client. The problem went away.
View ArticleChanging location of binlog prevents starting up
I've copied data/binlog files between MariaDB (10.3) instances and wanted to change the location of data/log files on the new machine. I adjusted the paths in the my.cnf file. On start on the new...
View ArticleTruncate temporary table commits transaction?
Is this correct that a "CREATE TEMPORARY TABLE IF NOT EXISTS.." does not commit current transaction (I like that), but a TRUNCATE TABLE, which is temporary, does implicitly commit any transaction?Does...
View ArticleWhen range partitioning, should I drop primary UUID-like key?
I'm aware of the limitation "every unique key on the table must use every column in the table's partitioning expression". So given that my primary key is a UUID, it's going to require a...
View ArticlePoint in time recovery with mariabackup
I have a full backup dir and a bunch of hourly (incremental) backup directories taken with mariabackup, which is a port of the XtraBackup. How does one restore the MariaDB up to a specific minute (or...
View ArticleWhy do I have two collations in mysqldump, under stored procs?
When I run mysqldump --routines I'm seeing multiple collations for my stored procs. Neither my DB nor columns use utf8mb (I understand the benefits of utf8mb over utf8 but let's leave it for...
View Article