Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature](backup) backup cooldown data #45886

Open
wants to merge 2 commits into
base: branch-2.0
Choose a base branch
from

Conversation

justfortaste
Copy link
Contributor

@justfortaste justfortaste commented Dec 24, 2024

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Currently, backup/resore is not supported for cooldown data.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@github-actions github-actions bot added area/planner Issues or PRs related to the query planner kind/docs Categorizes issue or PR as related to documentation. kind/test labels Dec 24, 2024
@justfortaste
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 49645 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5e4054540cd550f66bbef6e277e63c01569ca9e5, data reload: false

------ Round 1 ----------------------------------
q1	18286	4555	4442	4442
q2	2076	162	149	149
q3	10465	2013	1957	1957
q4	10317	1252	1318	1252
q5	8382	4006	3962	3962
q6	236	126	124	124
q7	2061	1620	1606	1606
q8	9541	2740	2723	2723
q9	13850	10017	10161	10017
q10	8673	3495	3483	3483
q11	409	256	260	256
q12	479	306	309	306
q13	18370	3992	4043	3992
q14	356	327	328	327
q15	508	468	460	460
q16	540	464	473	464
q17	1135	981	995	981
q18	7205	6848	6959	6848
q19	1729	1585	1505	1505
q20	523	322	313	313
q21	4480	4107	4084	4084
q22	493	394	420	394
Total cold run time: 120114 ms
Total hot run time: 49645 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4383	4420	4388	4388
q2	323	228	219	219
q3	4176	4133	4158	4133
q4	2739	2753	2745	2745
q5	7242	7175	7225	7175
q6	240	124	122	122
q7	3275	2849	2871	2849
q8	4370	4501	4519	4501
q9	14261	14089	13934	13934
q10	4267	4316	4274	4274
q11	750	678	734	678
q12	1024	859	868	859
q13	7116	3743	3727	3727
q14	450	435	424	424
q15	508	456	469	456
q16	639	603	595	595
q17	3781	3871	3904	3871
q18	8881	8824	8732	8732
q19	1712	1705	1647	1647
q20	2399	2137	2130	2130
q21	8499	8493	8387	8387
q22	1001	950	966	950
Total cold run time: 82036 ms
Total hot run time: 76796 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 215113 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5e4054540cd550f66bbef6e277e63c01569ca9e5, data reload: false

query1	944	433	388	388
query2	6537	2400	2133	2133
query3	6929	199	198	198
query4	23480	21687	21844	21687
query5	19740	6682	6679	6679
query6	299	226	247	226
query7	4306	313	313	313
query8	251	270	241	241
query9	3102	2670	2595	2595
query10	462	346	314	314
query11	15551	15082	14903	14903
query12	131	80	75	75
query13	1029	456	436	436
query14	17936	13603	14148	13603
query15	368	235	235	235
query16	6489	286	257	257
query17	1792	962	924	924
query18	901	331	332	331
query19	223	164	154	154
query20	111	106	103	103
query21	195	103	98	98
query22	5253	5046	4940	4940
query23	34564	33498	33696	33498
query24	7919	6543	6486	6486
query25	563	440	421	421
query26	1275	174	172	172
query27	2445	307	303	303
query28	6091	2308	2285	2285
query29	2949	2773	2705	2705
query30	252	175	175	175
query31	1006	792	798	792
query32	72	64	56	56
query33	459	272	263	263
query34	939	507	486	486
query35	1145	977	967	967
query36	1299	1126	1484	1126
query37	174	62	64	62
query38	3116	3003	3021	3003
query39	1394	1339	1341	1339
query40	308	101	97	97
query41	41	40	38	38
query42	94	88	88	88
query43	633	649	623	623
query44	1215	720	718	718
query45	255	233	237	233
query46	1352	1046	1038	1038
query47	1993	1748	1898	1748
query48	513	417	423	417
query49	682	380	393	380
query50	975	668	661	661
query51	4860	4722	4669	4669
query52	96	87	91	87
query53	243	195	195	195
query54	2829	2612	2612	2612
query55	90	91	87	87
query56	218	216	207	207
query57	1323	1103	1261	1103
query58	228	211	195	195
query59	3565	3562	3337	3337
query60	226	213	222	213
query61	99	97	99	97
query62	839	479	473	473
query63	207	176	179	176
query64	3572	1656	1526	1526
query65	3732	3653	3636	3636
query66	782	413	408	408
query67	16095	15201	15267	15201
query68	9123	660	679	660
query69	497	294	271	271
query70	1612	1378	1383	1378
query71	402	345	310	310
query72	6805	5006	5034	5006
query73	815	327	325	325
query74	6303	5815	5914	5815
query75	4686	3765	3773	3765
query76	4691	1284	1340	1284
query77	625	262	260	260
query78	12878	11899	11786	11786
query79	6830	651	657	651
query80	2727	421	409	409
query81	519	254	251	251
query82	1739	102	107	102
query83	189	134	132	132
query84	259	74	73	73
query85	1381	329	333	329
query86	370	286	318	286
query87	3275	3087	3030	3030
query88	4858	2325	2334	2325
query89	437	304	301	301
query90	1778	215	212	212
query91	163	133	139	133
query92	63	52	53	52
query93	5291	578	566	566
query94	895	215	206	206
query95	2073	2043	2088	2043
query96	706	330	337	330
query97	6621	6480	6487	6480
query98	236	215	208	208
query99	2859	899	911	899
Total cold run time: 320462 ms
Total hot run time: 215113 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.5 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5e4054540cd550f66bbef6e277e63c01569ca9e5, data reload: false

query1	0.03	0.03	0.02
query2	0.07	0.02	0.02
query3	0.25	0.05	0.05
query4	1.75	0.08	0.08
query5	0.54	0.53	0.53
query6	1.27	0.63	0.66
query7	0.01	0.02	0.01
query8	0.03	0.02	0.02
query9	0.52	0.48	0.47
query10	0.54	0.54	0.54
query11	0.13	0.09	0.09
query12	0.12	0.10	0.09
query13	0.62	0.61	0.61
query14	0.79	0.78	0.79
query15	0.78	0.76	0.77
query16	0.37	0.39	0.39
query17	0.98	0.97	1.02
query18	0.22	0.25	0.25
query19	1.96	1.80	1.88
query20	0.02	0.01	0.01
query21	15.49	0.57	0.55
query22	2.36	2.65	1.52
query23	17.52	1.13	0.81
query24	6.19	0.77	1.17
query25	0.36	0.11	0.05
query26	0.61	0.15	0.16
query27	0.05	0.03	0.03
query28	7.00	0.78	0.79
query29	12.64	2.31	2.37
query30	0.58	0.53	0.54
query31	2.81	0.39	0.38
query32	3.36	0.50	0.49
query33	3.11	3.07	3.08
query34	15.25	4.84	4.83
query35	4.84	4.89	4.86
query36	1.07	1.00	1.03
query37	0.06	0.04	0.04
query38	0.03	0.02	0.02
query39	0.02	0.02	0.02
query40	0.16	0.15	0.14
query41	0.07	0.02	0.01
query42	0.02	0.02	0.01
query43	0.04	0.01	0.01
Total cold run time: 104.64 s
Total hot run time: 30.5 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 5e4054540cd550f66bbef6e277e63c01569ca9e5 with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       22.8 seconds inserted 10000000 Rows, about 438K ops/s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/planner Issues or PRs related to the query planner kind/docs Categorizes issue or PR as related to documentation. kind/test
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants