Official records say

The Official Records constitute the most extensive collection of primary sources of the history of the war. They include selected first-hand accounts, orders, reports, maps, diagrams, and correspon...dence drawn from War and Navy Department records of both Confederate and Union governments.Official Records, Series I, Vol XVI Part I, pg. 805, Lt. Col. Parkhurst's Report (Ninth Michigan Infantry) on General Forrest's attack at Murfreesboro, Tenn, July 13,1862: "There were also quite a number of Negroes attached to the Texas and Georgia troops, who were armed and equipped, and took part in the several engagements with my forces during the day." Official Records, Series 1, Volume 15, Part 1, Pages 137-138, report of the Union commander: "Pickets were thrown out that night, and Captain Hennessy, Company E, of the Ninth Connecticut, having been sent out with his company, captured a colored rebel scout, well mounted, who had been sent out to watch our movements." Official Records, Series I, Vol. XLIX, Part n, pg. 253 - April 6, 1865 "The rebels [Forrest] are recruiting negro troops at Enterprise, Miss., and the negroes are all enrolled in the State." Official Records, Series I, Vol. XIV, pg. 24, second paragraph, Colonel B. C. Christ, 50th Pennsylvania Volunteer Infantry, official report of May 30, 1862, Pocotaligo, SC., "It is also difficult to state the force of the enemy, but it could not have been less than from 600 to 800. There were six companies of mounted riflemen, besides infantry, among which were a considerable number of colored men."
Official Records, Vol. XIII, Chapter XXV, pg. 688 - "...We are not likely to use one negro where the rebels have used a thousand. When I left Arkansas they were still enrolling negroes to fortify the rebellion." - September, 1862

Official Records, Correspondence, Etc., Vol. II, pg. 218 - "...they [the Confederacy] have, by means of sweeping conscription, gathered in countless hordes, and threaten to overwhelm the armies of the Union, with blood and treason in their hearts. They flaunt the black flag of rebellion in the face of the Government, and threaten to butcher our brave and loyal armies with foreign bayonets. They arm negroes and merciless savages in their behalf." - July 11, 1862 - Rich D. Yates, Governor of Illinois

Official Records, Vol. XIX, Chapter XXXI, pg. 617 - Record of the Harper's Ferry Military Commission (U.S. Army)
Question. "Do you know of any individual of the enemy having been killed or wounded during the siege of Harper's Ferry?" Answer. "I have strong reasons to believe that there was a negro killed, who had wounded 2 or 3 of my men. I know that an officer took deliberate aim at him, and he fell over. He was one of the skirmishers of the enemy[Confederate, ed.], and wounded 3 of my men. I know there must have been some of the enemy killed." Question. "How do you know the negro was killed?" Answer. "The officer saw him fall." Official Records: Series 2, Vol. 6, Part 1 (Prisoners of War) p. 17-18 - "...before one single negro or mulatto was mustered into the U.S. service you had them organized in arms in Louisiana. You had Indians and half-breed negroes and Indians organized in arms under Albert Pike, in Arkansas. Subsequently negroes were captured on the battle-field at Antietam and delivered as prisoners of war at Aiken's Landing to the Confederate authorities, and receipted for and counted in exchange."
Official Records, Vol. XLI, Chapter LIII, pg. 670 - PATTERSON, [November] 24, 1864 - "Colonel MAUPIN: I have arrived with my squad on return. Captain McClanahan has gone on the upper road for Pilot Knob; will all arrive there to-morrow. No rebel force below. We have turned up eleven bushwhackers to dry and one rebel negro. No man hurt on our side. The men are generally well."

Official Records, Series 1, Volume 4, p.569 - Report of Colonel John W. Phelps, First Vermont Infantry: CAMP BUTLER, Newport News, Va., August 11, 1861 - "SIR: Scouts from this post represent the enemy as having retired. they came to New Market Bridge on Wednesday, and left the next day. They-the enemy-talked of having 9,000 men. They were recalled by dispatches from Richmond. They had twenty pieces of artillery, among which was the Richmond Howitzer Battery, manned by negroes. . . Their numbers are probably overrated; but with regard to their artillery, and its being manned in part by negroes, I think the report is probably correct. "

Official Records, Series 1, vol 35, Part 1 (Olustee), Page 442-443, S.C., FLA., AND ON THE GA. COAST. Chapter XLVII - Report of Bgen Asboth, USA: "...when I proceeded to Milton, Fla., a distance of 9 miles, and after rebuilding the destroyed bridge on the Arcadia Creek, I came upon the enemy, about 100 strong, and consisting of Captain Goldsby's (Alabama) cavalry company and a new militia infantry company, mounted...Having received early information of the arrival of two army steamers at Bayou Mulatte, the enemy had sent his stores on seven wagons in time toward Pollard, and seemed prepared and decided to accept a fight in the camp at the upper end of the town, but fled, upon our impetuous charge, in all directions. We pursued them closely for 7 miles, and captured 4 privates of Goldsby's company and 3 colored men, mounted and armed, with 7 horses and 5 mules with equipments, and 20 Austrian rifles."

Official Records, Series I, Vol. XVII, Chapter XXIX, Pg. 635-637 - December 28, 1863 - "...It had to be prosecuted under the fire of the enemy's sharpshooters, protected as well as the men might be by our skirmishers on the bank, who were ordered to keep up so vigorous a fire that the enemy should not dare to lift their heads above their rifle-pits; but the enemy, and especially their armed negroes, did dare to rise and fire, and did serious execution upon our men...The casualties in the brigade were 11 killed, 40 wounded, and 4 missing; aggregate, 55. - Very respectfully, your obedient servant, D. STUART, Brigadier-General, Commanding"

Official Records, Series I, Vol. III, Correspondence, etc., pg 767-768 - "CAMBRIDGE, September 4, 1863. His Excellency A. LINCOLN, President of the United States: "...excitement here growing out of the recruiting of colored troops, and as some of the recruiting officers are acting rather indiscreetly, I fear, by taking slaves in their recruits, and the slaves of loyal as well as disloyal persons...to enlist slaves as well as free people is creating a great deal of anxiety among the people...we ought to use the colored people, after the rebels commenced to use them against us.